This mail was originally posted to the Arisia Corporate list in October, 2007. I've trimmed out some things that are not relevant to TF, but the bulk of the information should be useful.
Since this document was written, the following changes have occurred:
I've heard through various sources that there is an item on next week's agenda regarding Arisia hosting, as well as some grumbles throughout the community. I won't be able to attend the meeting, but as the one who is probably the most tightly connected with Arisia's hosting configuration, I felt it might be appropriate to send out some informational mail so those discussing the topic have accurate and current information regarding not only our hosting technical details, but areas of responsibility as well.
This is a long mail, if this is a topic you're interested in, and are going to be participating in the discussion as well as the decision process, then I recommend a cup of coffee and 10 minutes or so to read through.
Any discussion revolving around Arisia's online presence really needs to take into account what we have now.
Arisia's corporate and convention online resources take in 5 different servers, placed in 4 different locations.
1) msb.ernest-doss.org - (formerly known as 'msb.significant.com'). msb is owned by Dwight Ernest, and is a dedicated host, colocated at ServerBeach?. We've never physically seen the host, but we have full administrative control over it. It is an AMD XP 2100+ running Redhat Enterprise Linux ES Release 3. Note that this is a commercial hosting service, and is being paid for by Dwight and other members of the support team.
2) boomer.homeport.org - Owned and operated by Dave Belfer-Shevett. It is a AMD Sempron 2400+ 1U rackmount server running Ubuntu Edgy Eft linux. It is located at RNK Telecom in Bedford.
3) msb2.ernest-doss.org - Owned and operated by Dwight Ernest. msb2 will be the replacement host for msb. It is a rackmount Compaq server Dual PIII-1.2gig server with a RAID disk array. As of 10/15/2007, only basic setup has been done, no services have moved to it yet.
4) two-step.netbusters.com - Owned and operated by Alex Latzko. I don't have full details on this machine (and I don't seem to be able to log into it) ('''update 2/5/2009: two-step is an Ubuntu 'Gutsy Gibbon' Linux machine, running on a Celeron 2.6gig machine with 1gig of memory and 120gig of disk space. It is colocated at Conversant in Marlborough, MA )
5) ns.netscum.com - A host in California that provides secondary DNS services.
(For amusement, this is a picture of boomer and msb2) http://www.flickr.com/photos/planet-geek/512287079/
Each of these machines has specific functions hosted on them, and they have different purposes. I'll keep my description of what each of these machines is doing to Arisia-specific details. Note that boomer, two-step, and msb are heavily used servers for a variety of purposes other than Arisia.
Until 2 years ago, msb hosted 99% of Arisia's functionality. All websites and all mail services came through it. When we moved to the Drupal CMS platform, we moved the per-convention hosting to boomer. www.arisia.org still points to msb, but it is an immediate redirect to 2008.arisia.org, which is hosted on boomer. (dbsnote 2/9/2009 - 2009.arisia.org is also pointing to boomer)
msb holds all of Arisia's mailing lists. It also holds the archives of convention websites from 2006 and earlier.
On the corporate side, old Mentor issues are all stored on msb.
boomer has been taking on more of the load of managing Arisia sites and resources. The 2007 and 2008 websites are hosted on boomer, as well as the corporate site. DNS services are also hosted there. boomer also runs 'irc.arisia.org'. Also on boomer is the cvs repository that holds the source code to CONGO as well as Zambia.
arisia.stonekeep.com (the CONGO host), as well as Arisia's registration database (providing attendance data going back twenty some odd years) lives on boomer.
two-step has two primary purposes. First is hosting the techno-fandom wiki, which is used by many Arisia departments for organization. Second, it is the Zambia server host for programming.
two-step is managed directly by Alex, though occasionally I lend a hand. I have very little input into true system maintenance, upgrades, or support, that is all handled by Alex.
boomer and msb are managed by two groups of 'geeks', with about an 80% overlap between them. The primary maintainers are myself, Dwight Ernest, and Tim Pierce. In general, I take primary responsibility for boomer, and Dwight takes primary responsibility for msb. This has resulted in varying levels of service and response. With boomer as my primary machine, I'm naturally more inclined to 'pay more attention' to it. Dwight, as primary shareholder of msb, is ultimately responsible for that machine's operation.
For the most part, this balance of support works. On occasion we get lapses where the primary stakeholder is not available to 'drive' a problem through. This is what happened in September when we had the mail outage on msb. Dwight was unavailable to work through the problem, Tim and I stepped up to the plate and after approximately 6 hours of work, managed to track back the problem and get things running again.
Dwight's schedule has gotten more complex, so Tim and I are ramping up our involvement in the management of the servers. For my part, I am 100% committed to booomer's uptime, maintenance, and security. I am ultimately responsible for it's stability, and I owe it to my users (there are around 40 active shell users on boomer, and perhaps 50 websites) to put in my best effort to keep the machine up, stable, updated, and to respond to user requests. For the record, I also run boomer's backups regularly (copied offsite, then burned to DVD).
The current configuration is not steady-state. We are constantly adding websites, changing things, and working on new configurations. We have already done one major OS upgrade on boomer (moving from Dapper Drake to Edgy Eft about 6 months ago), and we've upgraded the RAM in boomer (from 512meg to 2gig).
msb is in need of maintenance and upgrading. This is the primary purpose of msb2, though the migration to msb2 has stalled somewhat. Tim and I are re-starting the process now. msb is currently suffering from poor performance and periodic 'long delays' in either web servicing or mail gating (we have fairly severe spam-filtering on msb and boomer due to the immense volume of mail both machines service). We are also quite unhappy with ServerBeach? as a colocation service. For the amount of money being paid to them (around $150 a month), the level of service were receiving is sub-par. Another reason to migrate.
At the moment, RNK Telecom in Bedford (where Dwight works) is perfectly willing to host 1-2 machines for us. They have phenomenally good network connectivity, and in the 2 years boomer has sat there, we have had 2 network outages, both lasting less than 30 seconds. boomer was offline twice beyond that, once when a network cable worked itself loose from the switch (I drove over to fix it), and another time due to an inexplicable system lockup (that was 2-3 months ago, and has not repeated).
We plan on continuing hosting boomer (and msb2) at RNK until situations arise where that is no longer feasible. We are working on a formalized agreement with RNK so the hosting is 'business-sanctioned' and won't be randomly disabled (a very very very small chance of this happening, but we're still being cautious).
We also plan on migrating services off of msb as quickly as possible. Some moving to boomer, some moving to msb2.
Up until now this has been background information. It is the state of how things are. Now I'd like to talk about where we should be going. The discussion as I understand it revolves around "How do we consolidate Arisia's online resources so we're not so spread out"
In general, I agree this is a very good idea. The machines are too spread out now, and there are too many 'fingers in the pie' as it were. If we have a failure, where do we go for help?
Here's my list of suggested moves.
1) Repoint 'www.arisia.org' to boomer. 2007.arisia.org and 2008.arisia.org are already there, and 2009 will be there as well. Right now msb is acting as a 'reflection' point, requiring it to be online for general users to make it to 2007.arisia.org. This was a matter of convenience when we first went online with the (year).arisia.org structure, and now we should go the next step and point www.arisia.org directly to boomer, with the appropriate redirect there. Website uptime then will only depend on one machine, not two.
2) Complete moving all old sites and Mentor files to boomer, and repoint the rest of arisia.org resources. Boomer has plenty of disk space and bandwidth to handle this, and the old sites and mentor files are very very low impact. Consolidate.
3) For the moment, do not move the mail system off msb. These are too critical a component for Arisia to be messed with this close to a convention. However, after the 2008 event, migrate the mail aliases to a managed mailing list structure such as Mailman. This can be on msb2 or on boomer, it doesn't matter.
4) There's been some question about why Zambia is not on boomer as well. Zambia needs data imports from CONGO regularly anyway, we're not sure why it's split. We have a wiki 'farm' configuration on boomer that consolidates many many hosted wikis (about 20) into a single code base (making upgrades and maintenance very easy). I can move the Arisia wikis as well as Zambia to boomer. Again, I don't think moving Zambia this close to an event is wise, so this should probably wait until after con-time. Also, the wikis are not as mission-critical as many of the other resources. Alex will need to talk to this about how two-step is being managed, but I'll just say that boomer has plenty of bandwidth (both computationally as well as network wise) to handle this.
Many of my conclusions and suggestions will come across as ego-centric and overreaching. "I'll run everything for you!" I acknowledge this. However, I have been working very closely with the webmasters / webmistresses in the building of all the online content, and have been responsive and responsible when there have been problems and questions. I also am not trying to take this on alone. I cannot run all of Arisia's online content by myself. Cris Shuldiner is doing an awesome job of maintaining the mailing lists and servicing user requests. Lisa Holsberg is managing the website and keeping content up to date. The reg team is maintaining the online registration database. Phi is a regular data-miner within the CONGO database instance, and is regularly doing maintenance on the data. The Zambia team keeps Zambia up to date and services requests constantly.
The reason I suggest making greater use of boomer is that we have a fantastic team even beyond the Arisia group. arisia.org is one service of many that boomer provides, and we have a half dozen admins on boomer, who are using the machine constantly, that respond to problems and service requests day in and day out. Tim, Dwight and myself are three of them, we also have David Mortman, Adam Shostack, Ben Cordes, Noel Rosenberg, and Duncan Hill - some of these people are very active in the Arisia community, some are not, but all are familiar with the host, it's configurations, and it's quirks. Some of these folks also maintain msb, but we tend to focus on boomer.
If Arisia corporate decides that boomer, msb, and two-step are not appropriate resources, and wish to 'roll a new one', I will help with the migration off, but I feel unless we have a group of technical folks familiar with Arisia operations and that have the personal time and energy to commit to a long term maintenance prospect (and I've been working with Arisia hosting for something like 6 years now), then moving to a new host or a new platform or a new environment would be difficult, if not damaging to the online membership.
Feel free to contact me with any questions, comments, or updates. I will be travelling Thursday through Sunday, then in a meeting Sunday afternoon and early evening.
-dbs
_______________________________________________ Corp mailing list Corp@arisia.org http://arisia.org/mailman/listinfo/corp