Zoto is down for now.  Back toward the middle of last year we started losing drives out of both the NAS and the cluster boxes.  These drives were manufactured by Western Digital. Evidently they don't fare so well, and have a high failure rate.  Lesson learned: don't buy WD drives. Anyway, the site went down about 10 times over about half as many months, with Clint (without who I could not have done all this) and I trying to keep things up the best we could.  In September I ended up moving Zoto to my house in California (where they have lots of bandwidth) and continued nursing it along.  Toward the first of the year the NAS crashed again, this time with a fatal two drive failure.  Oh noes. You can read about my adventures in getting all the photos back off a multi-drive failure of a RAID5 array with (12) drives in it, and trying to get Zoto migrated to Amazon on this blog.  The tl;dr; of the whole mess is that midway through the process I made the decision that I didn't want to continue to risk the possibility of losing more people's photos by continuing the service in such a flaky manner.  And so, regardless of a few crackpot's protests, I decided to leave the site down permanently and focus on getting peeps photos back. I've been successful at getting photos back to the people that need them.  By doing that, hopefully I salvaged what small bit of credibility I had in promising Zoto would "keep your photos safe".  However, I've probably lost all credibility in my ability to hack up a horribly complicated system to run on Amazon.  Can't win them all! If you are in need of recovering your photos from the old Zoto install, please see this form:  http://kordless.wufoo.com/forms/zoto-accounts/.  I run the updates every few days.  Depending on demand, I may automate the process with a self serve form. If you are interested in what I'm planning on doing with the site and company, stay tuned.  It should be relevant to your interests, and be well within my ability to implement. Permalink | Leave a comment »

The code responsible for uploading and notifying you of your archive links is now running.  Anyone who filled out the request form and requested a photo archive dump in the last few months will be receiving a link via email sometime in the next few days as your archive is uploaded and processed. If you haven't filled out the form, or filled it out and didn't request a photo archive but want one, post here and I'll add you to the list of accounts being processed. I'm actually not sure how long this entire process will take - perhaps a couple of days at most.  It's uploading as fast as my connection at the house can stand! Kord Permalink | Leave a comment »

Been working on the export code for about 8 hours today and am happy to say the process of exporting photos is now underway.  The first step is to pull a given accounts photos out of the database, locate the images on the backups, then place those images in a directory strucutre that echos the dates and filenames from the original images.  The zip files are by year/month, so I'll be providing some tips on how to do bulk downloads and unzips to Windows or OSX.  Some images don't have the right date on them, so I'm doing best effort to put them somewhere in the directory structure. As soon as a few of these accounts are zipped, I'll start the second step of uploading the zips to the server on Amazon.  When the upload is complete you'll be emailed a link to download the image archives. I took the extra step of reporting how many photos you had, how big they are all in total, and a report of which photos (name, date, size) were lost in the drive crash.  I ran my account just now and I had 16 lost photos out of about 5,500 total.  Another account I just ran had only one photo lost, probably due to the fact he was a user from a later stage in the company. I'll keep you guys posted as this progresses.  Going to dinner.   Permalink | Leave a comment »

I'm working on getting the export code done this weekend to start emailing people a link to their photos for download.  If you didn't indicate you wanted an archive of your photos on the Wufoo form, I won't be emailing you this link. Upon futher consideration, and given the circumstances of where we (you guys and me) are, I think it's best to just shut things down. All in, only 114 people have filled out the form to either get a refund, export their photos, or get their accounts back.  While I've been partially successful at bringing the site itself back online, I have been unable to get the database exported and moved over.  The machine the database is on is now the last box left from the Zoto cluster that is still operational.  I'm worried that if I ran the service off it, it would fail and bring us down again.  I could try to do a partial export, but chances are that I'll fuck things up if I do.  Chances are I'll run into development roadblocks that I don't have time for. A few angry people have emailed me about how long this has taken and what "we" are doing about it.  First, there is no "we" here, it's just me, Kord, sitting around on the weekend trying to fix this stuff.  Yeah, over the last 2 months I've not been working on it much due to my work schedule, but I am still intent on getting photos back to people who need them.  I'm not making money off this people, quite the contrary, but I am still keen on doing what I can to make things right - no matter how fast you think I should do it. I promise you don't want me bringing the site back up at this point.  I wouldn't have time to work on it anyway, and it's just a big source of irritation for everyone.  I may rework the site into a Twitter image hosting service though, as the name works well for that, and such a service would be like 20x easier to maintain.  You have no idea how complicated this code is - it's a fucking nightmare to work with all it's dependencies. Anyway, I've prattled on enough.  Sorry for all the dead air.  Trying to get this done finally.

Kord   Permalink | Leave a comment »

Last night the last dump of photos from the old drives was completed. After two months of effort, I've been able to recover 99.9% of the photos that were on the system. I'm in the process now of making a complete backup of the recovered photos, and I'm ready to begin work on getting the proper photos moved over to production to bring the site back up. Backups of these photos will also be available for raw download in a few weeks time - it'll take a while to process everyone. Percentage-wise, if you had a thousand photos on the system, chance are you lost one of them. I'd say that's better than losing all of them. Anyway, I'll keep you guys posted. The worst is behind me now. Won't be long to get things back to "normal". Cheers, Kord Permalink | Leave a comment »

About 2 weeks ago, midway through copying photos, I lost yet another drive out of the array on the NAS. So far, during this whole ordeal, I've lost a total of 4 drives out of the NAS. Thankfully, I have complete block-level backups of all disks in the array, so I was able to rebuild the disk with a little swapping around of cables in the NAS. To give you an idea of the mess involved, I have 11 cables running out of 3 cards on a borrowed PC snaking into the NAS chassis. I couldn't install the controller cards in the NAS because the motherboard didn't have enough PCI slots to accomidate them. The problem with this approach is that I'm deathly afraid of disconnecting the cables run because the RAID recover software from Runtime depends on the drive order to work correctly. This makes it hard to swap back in the backup drives to recover and rewrite the backup images. As I type this, the copy has be restarted, and is underway again. I'm starting from the beginning, but telling the system to to ignore existing files. When I get about 1/4 of the way through, I'll swap back to copying all files, due to the failure of the other drive causing a few photos to be corrupted. I'd like to ensure all data is recovered normally here. This ordeal will soon be over, and I aplogize again for leaving the site down for so long. I felt that, in the event of complete recovory failure, that I would be well to leave it down permanantly. If, and I'm giving it good chances now, I'm able to do a full recovery, we'll be back up again sometime soon. Cheers, Kord Permalink | Leave a comment »

Well, after a long month of data recovery, I'm finally seeing intact photos from the drives! This has been an enormous undertaking, and I've honestly not done anything as difficult as this before. In theory, I should be able to start uploading photos to the site this week. I still have to do a "recovery" copy to a set of backup disks before I start uploading, but right now I'm browsing the file system and looking at intact photos, so I think that should go fairly quickly. Here's a partial breakdown of what had to take place to get where I am today: on NAS failure, tried to discover which drive failed first (about 2 days) tried doing a controller board swap to reactivate drives (about a day) sent suspected last drive off to get recovered (almost 3 weeks) on getting the drive back, discovered it was the first failed drive (which means the array won't reassemble) do research on software/solution that will provide recovery from a broken (yet intact) array decide that backing up current drives is required and an imminent priority began backing up old (11) 500GB drives to new compressed drive images, one at a time (about a week) figured out I could have copied faster with another server I had at hand (damn) discovered making compressed drive images was a bad idea if i was going to do a virtual reassemble (double damn) assembled a new RAID 0 array from a bunch of random drives I had at hand (about 2 days) copied drive images yet again to new RAID partitions (two at a time, very fast, about 2 days) set new backups aside and put old drives back in NAS found out the controller card I had didn't support direct drive access went to Fry's and bought (2) controller cards to give me (11) total SATA ports discovered the NAS didn't have enough slots for said cards stole my son's gaming rig to install cards and access drives installed windows 7, then found out recovery software won't run on it installed XP (thought I'd never do that again) installed about (3) different pieces of software to try out ran one for 3 days in "filesystem discovery mode" before deciding it wasn't working installed software from runtime.org and then it told me I needed to scan and send in data called runtime and open-e for tech support again to double check my approach started runtime scan (took about 5 days to scan all drives) upgraded my cable to 8mbps upload in preparation for sending photos to AWS uploaded runtime scan (about a day) waited on results (about 4 days) installed yet another piece of software that reads results and finds an XFS filesystem (about a day) PHOTOS FOUND OMG So I still need to copy stuff over, and then figure out a way to upload the photos that have been requested. This will take AT LEAST another few weeks to fully complete. There will assumably be issues in doing this! In the meantime I will activate all the paid accounts on the new servers and turn the site back on. I was waiting to do this until I knew photo recovery was possible. Now I do, I see no reason to not bring the site back up. I'll keep you posted. Kord Permalink | Leave a comment »

Short update on the work done this weekend. Things have progressed slowly because of the difficulty in installing the software on an Amazon servers. I finally found a good server image that basically emulates the older servers I was running on the dedicated Zoto servers. Most of the software is running correctly now, including the uploader! With the exception of not having user account info loaded, I think most of everything is working correctly. I should have the site active for paid users sometime later this week. Seems pretty snappy too! I've forwarded all traffic to the main page to the blog for now. I've also disabled signups until things get better, and those will remain disabled until I can get all the details (like emails) sorted. The recovery of the photos are another story. The drive that was recovered by the service ended up being the degraded drive, which sucks. With it being degraded, I can't immediately boot up the array for backup. Instead, I'm having to copy each drive, one at a time, to a set of backup disks so they can be reassembled for recovery. It takes about 8-9 hours per 500GB drive, with a total of 11 drives needing to be backed up. In theory, this process will be done sometime tomorrow afternoon. Backups good. The problem with this approach is that I need to copy all these images back again to the NAS to rebuild them correctly. This will take even more time, so it's unlikely that I'll have any photos recovered before the end of the week. If I had to guess, I'd say end of the month for sure, but not sooner. Recovery of a broken RAID 5 array is possible with the data I have, just slow. I'm still about 95% certain all the data can be recovered, FWIW, so nobody should be worried yet. I'll keep you posted on the recovery. This is 3 weekends in a row I've grinded out time for this. The wife is tollerating it, but not for much longer. :) Kord Permalink | Leave a comment »

Bit of confusion as I'm trying to get the site working again - have to do it live folks. The recovery form I had up on the main page is now here: http://kordless.wufoo.com/forms/zoto-accounts/ Kord Permalink | Leave a comment »

Signups are disabled until I can complete the work needed to fix uploading photos. I'll leave the signups disabled until I can get the paid accounts who have requested it moved over. This process will take place in two steps, with the first step being an import of the account itself, followed by the second step of importing the photos. Getting there. Lots of moving parts here. Permalink | Leave a comment »

Whew. The drive recovery company just called and told me they recovered almost the entire drive, save for about 500KB or so of the data. The last recovery I did on the failed drive in September recovered all but 100MB of data, and only ended up affecting ~60 photos total out of about the 7 million we have on the system (this includes thumbnails). I would expect only a few photos to be affected in the recovery, so it's unlikely anyone will lose anything. Next step is to install the recovered drive and then immediately dump this data to a new array I've built while the drive has been out. All photos will be moved to this new array, and then from there will be used to import your photos to the new server on Amazon. One Amazon server has ~800GB of storage on it natively. I will utilize this space for photos and thumbnails (we make 10 for each photo) and then that drive will be duplicated onto what Amazon calls a "snapshot". My plan is to do daily rolling snapshots, and rolling weekly snapshots. That means there will be one instance of a photo on the server, a daily snapshot, and a weekly snapshot. Additionally, I'll be rebuilding the NAS at the house with newer drives, and will be rsync'ing the photos from the servers back to the house as a fallback. Each photo will be backed up 4 times. At some point the cost of storing free account photos will be more costly than I want to carry. I'll be implementing cleaning scripts to delete expired free accounts from the system to ensure these photos don't clutter up the system. It's usually a bunch of porn anyway. Damn freeloaders. Hang on - not much longer now! Kord Permalink | Leave a comment »

The recovery company is suppose to be swapping heads on the drive. I sent email yesterday for an update and didn't get any more than "we're working on it back". I looked, and I sent the confirmation to do the work in over a week ago, but the turnaround time is suppose to be 5-6 working days, so I guess they aren't that far off the mark, especially with the work they are doing - in a clean room even. Shouldn't be much longer. New Features

I've been wrestling with the uploader the past few days. Suprisingly it's been the hardest thing to get running on the new server, and it's really starting to piss me off. We used a code framework called Twisted when we wrote Zoto. Twisted has been upgraded a few times over the years and Zoto's older code uses some stuff that has since been removed. Anyway, the uploader client may or may not make an return at relaunch. As the current Flash uploader is also busted, I'll have to impliment a new one before you guys can upload stuff. Luckly I implemented a portion of one of our older uploaders and there are a bunch of drop in uploaders that work like our old Flash one, so it should be fairly straightforward. And you guys wondered why I never moved stuff to Amazon before! ;) One possible option for a "new" uploader would be the use of a product I've found called ExpanDrive. Basically you'd download/buy it ($40 though) and then use it to mount a directory on Zoto. You could drag and drop stuff in a local directory, and I could move/copy stuff out of this directory into your account. There's a free version of something similar for the Mac called MacFusion, which I use a bunch at work. Well, I have a bunch of ideas for stuff I can do once things get moved. It becomes SO much easier to work on the software once I get it off these older servers. Just a lot of work to get there. Hearty Thanks to You Guys

You guys have been really supportive through all this - especially seeing how I haven't really communicated well with you over the last few years. I've had about 80 people request their photos to be imported, yet not one refund request has been submitted. I really appreciate you guys making my life easier. Thanks a ton. Permalink | Leave a comment »

We've been in a lawsuit over the last few years. As we were a) not able to pay their licensing fees, nor b) willling to shut the site down, I ignored it. I was hoping it would go away, and just found out it did just that. This doesn't mean we can't be re-sued, but it's a good thing. Now if I can just get the website back up and running. I've bumped into a few problems with the uploader. Still waiting on the return of the drive as well. Permalink | Leave a comment »

Got the new server installed and running with all the newly updated software the server requires to run. Should have the site itself up tomorrow sometime. Will begin the process of installing paid account usernames on the new system, getting ready for the return of the hard drive, and images sometime next week. Signups will still work, but conversions to paid accounts (and taking payments) will be suspended until I can port the system to use Recurly. I fucking hate Paypal. Fixing Some Other Stuff Too

Still thrashing around on getting the uploader client to run. I'm in a better place to be able to fix the older Flash uploader - there are a ton of free ones out I'm looking at implementing. Oh, and I'll be ripping out printing as soon as the site comes back online. I think it was broken, but regardless printing goes. Most people didn't use it way back when, and there are a few patents that are seeking renumeration for sites like Zoto using "their" technologies. Whatever. Permalink | Leave a comment »

I've assembled a Wufoo form for handling requests from everyone: http://zoto.com/import.html This way I can better track and record who wants what. Last time this happened a few people slipped through the cracks due to me not having the correct Paypal email addresses for people. If I can't find the correct Paypal email address for you, I can't do a refund. Paypal only let's me refund people within the last 3 months or so with a transaction option. After that, I have to just issue a regular payment to your account. Without that account, and the original transaction, it's impossible to do the refund! For those of you looking to keep your account, or get a dump of your photos, just fill out the form and I'll process it as soon as I have the drive array back up. I expect to have the site up by the time the drive gets back from the recovery service, and you'll be able to log in and use your account, albeit without any photos. At that point you can start using the account again, upload photos, tag stuff, etc. If you request your older photo import, and have uploaded the same photo, the system will restore any meta data you had on that older photo. I'm sure there will be hiccups in all this process, but I'm doing the best I can to make it right. Kord Permalink | Leave a comment »

Drive is in the hands of the recovery guys currently. I'll update as I find out more from them in the next few days. It may be Monday before I know more. The outlook from them when I talked to them on the phone was that it was a bad read head, which is fixable in their book. We'll see. In the meantime I'm bringing up a new instance of Zoto on Amazon, with a temporary webserver already launched today to handle the inbound traffic. Getting the site up (sans photso) will take me a few days - mostly work in getting the server image built and configured. Photos will be stored on a RAID'd S3 instance, so the issues we've seen with our own NAS should become moot. Thoughts on migration are many. I'll probably have to write an import script to pull accounts, photos and tags from the older system. Again, I would transfer over all the info if it were possible, but it's just a nightmare to figure out what is needed, and what is not. There are a ton of older accounts on Zoto that aren't used, which are gumming up the works. Better to let everyone sort it out on their own. BTW, I've gotten a few emails from people (who are suprisingly chill with the whole ordeal) that they just want their photos and/or money back. That's my priority ATM to recover the photos and do refunds - will do more as soon as I have time. I appreciate the support you guys have given. Email for contacting me is zotohead@zoto.com. Kord Permalink | Leave a comment »

The bad drive has been shipped off to Dallas for attempted recovery. The guy I talked to at First Advantage said that it was fairly common for one or more of the drive heads in these WD drives to fail, even after a long period of operation, and he had seen instances of this type of failure (multiple drive at once) before. This doesn't bode well for using those drives much longer, as the remainder of them are all of the same ilk. I'm committed to taking whatever steps necessary to recover the drive, FWIW. One possible solution would be to configure a new system to import data (as needed) from the old one, allowing me to run the site on AWS where this sort of crap is much less likely to happen. BTW, wy apologies to the Posterous guys - looks like when I set the CNAME record for the blog I inadvertently sent people to their main page. Hasty work is shoddy work. Should be working now thanks to their help. I'll be trying to set a server up on AWS while the drive is off for repairs, and try to move this back to our normal format - not on Posterous. Kord Permalink | Leave a comment »

Still working on recovering drive array. Sending one drive off to recovery service to see if they can do a platter swap and recover. Confirmed theory that drives failed on a rebuild of the array - looks like one drive failed then the recovery started and killed the other drives. We're commited to trying all avenues at recovering the data, so please sit tight while we do this. I'll report back on any findings. Worst case it looks like a partial recovery can occur on the remaining disks, although there will assumably be data loss from that process, given the other drive is not recovered. Permalink | Leave a comment »

Just in case anyone thought this might be a tall tale! This is me trying to swap PCBs on the drives to see if I can fix one. Permalink | Leave a comment »

Last Thursday, the 15th of January, three separate drives in the Zoto NAS went south all at the same time - within an hour of each other. I'm still not certain what happened to them. About the only thing in common was the high humidity we've been having recently where I live - it's been raining non-stop for the last week. Apologies for taking so long to get the holder page up. Back Story

I've been having a rash of issues with the NAS going south at weird times of the day. Sometimes it runs for days, other times only a few hours before an error occurs. I've been trying to track it down, and have to assume these bad drives had a part to play in it. As a preemptive measure, I installed hot spares in the NAS to take over for a failed drive. The only problem with this approach is that it can't effectively combat a failure of multiple drives all at once. Best guess is that one drive failed, then the hot spare kicked on, causing the other two drives to spin up and then fail on their own. One of the drives was recovered on Friday, using a similar technique to what I did during the last outage. This time, however, it's worse in the fact two of the remaining drives are COMPLETELY dead. FWIW, all these drives are from the same Western Digital batch - all within a few serial numbers of each other. Not good. Back in October I was trying to move Zoto to Amazon, but failed in my attempts to migrate the database. We are running an older version of PostgreSQL and I, for the life of me, can't get the import to finish once I've exported the old database. I suspect it's a high cost on one of the insert queries that gets exponentially more expensive as time goes on - probably why some accounts (like mine) are slow to load on the site as well. Long and short of the move to Amazon was the cost of storing (and syncing) photos there was too high for us, and the technical hurdles too high. The power bill alone for Zoto nowadays runs about $200/month. While a few people still sign up for the service, it's definitely not paying for itself. Originally, both Rick and I wanted to continue running it for selfish reasons - our photos are on the site as well - and we had decided it was worth the cost and effort of it. However, and this is the difficult part, I think continuing to run it in this bastardized way is an injustice to all involved. I certainly don't benefit from it - quite the contrary - it' a HUGE fucking pain in the ass to maintain. I have a more than full-time job now, and simply have no time to dedicate to this thing that insists on breaking every week. Sometimes it's better to let a thing die. I've tried to give the service away a few times - notably to OU - but have met with no success on that front. I'm open to suggestions on finding a home for Zoto, hopefully with someone who will take care of it and maintain it. You can email me at zotohead@zoto.com if you have a good suggestion regarding donating the site to someone. TL;DR;

I'm working on finding a vendor to swap spindles on one of the remaining drives to try to do a recovery. Don't know how long that will take, or how much. We're going to pay the cost of doing this, and try to bring the site back up, but be aware we may fail in the attempt. Options at this point are: - If we can't recover the drive, we shut the site down, and try to do refunds (we have cash in Paypal to do this) - If we can't recover the drive, we move the service to Amazon, and start fresh. All paid accounts get a (2) year paid account, or some such deal. - If we recover the drive, get the site back up as-is, and then shut down signups. Tell everyone they have x many months to get their stuff off. - Find a good home for it. I'll donate the NAS and servers to the cause. Whoever takes it needs to be competent in maintaining Linux, databases, Python and JavaScript. That person will need to systematically replace all the remaining drives in the NAS which might fail at a later date. - Shut the darn thing off permanently. I'm sure apologies at this point are pointless, but you have mine anyway. This sucks!

Kord Permalink | Leave a comment »