Alternatively, join us on Discord.

+-Recent Topics

withgusto downsizing + migration (main server + ZNC)

Started by alberthrocks, March 18, 2016, 04:13:25 pm

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

alberthrocks

March 18, 2016, 04:13:25 pm Last Edit: March 19, 2016, 12:39:13 am by alberthrocks
First and foremost, I apologize for not posting this earlier! I completely forgot about this site when posting about this, so I really, really apologize for that! If you need more time to fill out the forms, just let me know!

If you are a former or current withgusto user (either with the shell services or ZNC), please read! We are moving and downsizing due to various circumstances.

If you happen to use ZNC, expect downtime starting at 9 PM EDT tonight! (RDNS downtime starting at 5 PM EDT as well - avoid reconnecting to IRC during that time.)

The details are really important - please read below for more info!

[spoiler=Main withgusto server downsizing + migration]
EDIT: Important Survey!
We are moving very soon - please fill out this IMPORTANT USAGE SURVEY by this Friday, 11:59 PM EDT!

Introduction
It has been a year since the last withgusto update.

About a year or so ago, a massive DDoS attack took withgusto down. This caused a "strike" to occur with the VPS hosting provider, and I temporarily closed withgusto's doors - specifically, hid it behind Cloudflare. This allowed Cloudflare to temporarily take the hit and reduce the load on the VPS. However, this had the effect of also blocking SSH, which was a lifeline into withgusto. Without a doubt, this is probably what drained significant use of withgusto.

Unfortunately, due to many commitments, I never got a chance to move withgusto. A year has passed, and withgusto is becoming increasingly difficult to maintain, both in terms of time and money. As such, regretfully, I must make the decision to wind down withgusto.

Note that this is NOT a shutdown - but we are planning on significantly reducing resources.

withgusto Downsizing
The proposed plan:

  • Reduce memory availability from 1 GB RAM to 128 MB - 256 MB


    • Significant memory usage is from clamd/amavisd/maild/spamd - around ~34.6%, or 354.3 MBs. mail@withg has been inactive for quite a while, mainly due to significant configuration issues. We are planning on removing this from withgusto, since mail services are difficult to maintain, and there is very little use of this service.

    • Deluge (web torrent client) takes around ~2.5%, or 25.6 MBs. We are planning on removing this from withgusto, since it's tough to maintain a client and not get into trouble for bandwidth/DMCA.

    • Current memory usage is 523 MBs. Subtracting all of these, 523 MBs - ~380 MBs = 143 MBs.

    • Depending on services people still want, and depending on how we shuffle things around, we may bring this down even further to <100 MBs. User programs and websites will fill the rest.


  • Reduce CPU cores from 2 to 1


    • We have rarely seen anyone use more than one CPU. Two CPUs is useful for compiling, but compiling doesn't happen too often - and VPS providers don't like constant CPU usage! Actual CPU speed will remain the same, just the number of cores will be reduced.


  • Use OpenVZ instead of KVM (less kernel control)


    • We have rarely used any features that would require a different kernel. All IPTables and advanced networking features are enabled in OpenVZ. Resources are allocated somewhat differently, but they should not have an impact on us.

    • OpenVZ is drastically cheaper than KVM (less VM). As such, this makes OpenVZ an appealing option going forward.


  • Disable backups for user data


    • At the moment, we pay for a backup service that we've (fortunately) never had to use. However, it incurs significant monthly cost, and as such, we will be phasing out automatic backups.

    • In the future, we may opt to use a free cloud storage service for backups to save money and achieve the same effect. However, we can NOT guarantee that this feature will arrive anytime soon - as such, you will now be responsible for your own data!

    • That said, before we move, user data folders (including /var/www/USER) will be backed up and offered to those who no longer wish to use withgusto.


  • Migration of user data ONLY - system will be cleanly installed


    • The current OS is very, very old - we're still using Debian 6, and the latest is Debian 8!

    • As such, we will be using Debian 8 in the future.

    • User data is easy to copy, but system data... not so much. If you have any system data that you want us to keep, let us know!



Future of withgusto Services
Service plan (some already mentioned above):

  • Mail service will be terminated. We don't see many (if any at all) using this, and this has been difficult to maintain. Moving forward, we will no longer host mail here.

  • Deluge (web torrents) will be terminated. Tough to maintain, and there are some other free/cheap options out there for your torrenting needs.

  • ZNC will be moved to a separate server/service from withgusto. We plan on running ZNC on a much lighter server - we've done some experiments, and it looks like it will work perfectly. It also allows us to improve stability, since withgusto is separate.

  • Omnimaga IRCd will be preserved - may either move to ZNC's location or withgusto - still deciding.

  • Depending on movability, user-installed services will be moved to the new server as well.

  • Other services will either be terminated or moved - we need to comb through them in the next few days!


withgusto Finances
Without a doubt, these are very tough cuts in service. The decision to cut these services was not taken lightly - however, in order for withgusto to continue (and not completely drop dead), we need to reduce these services and move to a lighter server.

Currently, I pay $15/mo for the main withgusto server, $15 for the backup service, $25/year for the backup.withg server, and around $16/year for the domain. Total yearly = ~$400/year!

With the new plan, I'm hoping to reduce this cost to something like this: $32/year for the main withgusto server, $15/year for the backup.withg/IRC/ZNC server, and around $16/year for the domain. Total yearly = ~$63/year... much more affordable!

Conclusion
All of this said, we still want your feedback! We don't want to move forward without hearing from you guys - if you don't like the things above, let us know and we'll try to accommodate! Since it has been a long time since we've had service, we want to hear from you in terms of how you will be using it.

Migration is NOT happening yet! Once we have some feedback, we'll announce dates for migration.
[/spoiler]

[spoiler=ZNC Migration]withgusto ZNC Migration (3/18/16, 9 PM - 12 AM EDT)
EDIT: Important ZNC Form!
As you may have heard, withgusto is undergoing a major downsizing to ensure viability in the future. As part of this plan, we are moving ZNC to a different, separate server to allow ZNC to operate independently of withgusto's main server, hopefully increasing the stability of both for times to come.

What does this mean for you?

  • There will be downtime for around three hours from 9:00 PM - 11:59 PM EDT this Friday, March 18th, 2016. ZNC will be completely disconnecting during this time. The downtime may be shorter depending on how things go, but don't count on it! If you maintain ops yourself, you should make sure to log in manually to keep ops during this time.

  • RDNS will be changed around 5 PM EDT this Friday, March 18th, 2016. Avoid reconnecting to IRC from ZNC during this time if you depend on hostname recognition!

  • After the move, your account and quality of service will NOT be affected. (In fact, we hope that it is MUCH better!)

  • We will be changing the login server once more. The hostname should stay the same - znc.withg.org. The only thing that will change is how you access ZNC. If you signed up on the form below, you will get an email soon on the change in login server. Please keep it private!

  • Since the ZNC service is essentially segregating from withgusto, we want to get user data with this survey! Please fill it out ASAP: http://goo.gl/forms/FehgMfKMp5

  • We will update everyone if there are any changes to this plan, as well as during and after the migration.


We apologize for this downtime, and as always, we thank you for flying withgusto! ✈

If you have any questions, comments, or concerns, feel free to ask away in this thread!

Updates:

  • [3/18/2016 10:14 AM] We will be changing the login server once more. If you signed up on the form above, you will get an email soon on the change in login server. Please keep it private!

  • [3/18/2016 10:14 AM] All preparations have been made for the move. Please get ready for the switch as soon as possible.

  • [3/18/2016 8:36 PM] Migration will occur in less than 30 minutes! Please backup ops for your channel, and get ready for the switch! Remember, the service will be down for around 3 hours, or until things are completely set up!

[/spoiler]

alberthrocks

Apologies for the double post! (A certain someone is permitting me to do it, just this once!)

@Juju @Streetwalrus @shmibs: I haven't gotten any response regarding ZNC usage - please fill that out as soon as possible! Thanks!

aeTIos

ceci n'est pas une signature

Juju

Read Zarmina!
YUKI-CHAAAANNNN
In the beginning there was walrii. In the end there will be walrii. All hail our supreme leader :walrii: --Snektron



if you wanna throw money at me and/or CodeWalrus monthly it's here

Streetwalrus

Quote from: Juju on March 18, 2016, 05:07:40 pm
I have an account on your ZNC?

Lol.
I don't really have a use for mine anymore. Do I need to fill out the form ?

alberthrocks

Quote from: Streetwalrus on March 18, 2016, 05:22:57 pm
Quote from: Juju on March 18, 2016, 05:07:40 pm
I have an account on your ZNC?

Lol.
I don't really have a use for mine anymore. Do I need to fill out the form ?


@Juju: yup you do! OmniNET only apparently...
@Streetwalrus: use the main server form and state that you don't want to use the services anymore!

DJ Omnimaga

March 18, 2016, 10:04:50 pm #6 Last Edit: March 18, 2016, 10:06:52 pm by DJ Omnimaga
Too bad you had to wind down some of the long-time Wihtgusto services albert, but I can totally understand why. On CW we tried to keep costs to a minimum because money doesn't grow in trees. To keep the site running we pay $12 a month (this includes a $2 backup feature that we could probably do away with if we had a second backup source besides @Streetwalrus (so if he's away for example we have an alternative or vice-versa), but in CW case it's made even harder by the fact that two of the staff are paid in Canadian dollars, but the site hosting is paid in US dollars. But basically the Minecraft server had to be hosted on a separate connection (currently running off Juju's internet at home).

I'm glad that Withgusto is staying alive in one form or another, though, but more importantly, I'm glad to see you around again in the community :walrii:


(Also yeah, @mentions only works if you have 20 posts, same for voting in forum polls and rating posts. Using the embedded shoutbox or sending PMs requires 5, though)

alberthrocks

Quote from: DJ Omnimaga on March 18, 2016, 10:04:50 pm
Too bad you had to wind down some of the long-time Wihtgusto services albert, but I can totally understand why. On CW we tried to keep costs to a minimum because money doesn't grow in trees. To keep the site running we pay $12 a month (this includes a $2 backup feature that we could probably do away with if we had a second backup source besides @Streetwalrus (so if he's away for example we have an alternative or vice-versa), but in CW case it's made even harder by the fact that two of the staff are paid in Canadian dollars, but the site hosting is paid in US dollars. But basically the Minecraft server had to be hosted on a separate connection (currently running off Juju's internet at home).
Yeah, $15/mo was pretty unsustainable for me... :( Interesting to hear $12/month though - which service are you guys using?

Quote from: DJ Omnimaga on March 18, 2016, 10:04:50 pmI'm glad that Withgusto is staying alive in one form or another, though, but more importantly, I'm glad to see you around again in the community :walrii:
Thanks! I'm probably going to be in and out, but I'll definitely be hanging out more on IRC!

Quote from: DJ Omnimaga on March 18, 2016, 10:04:50 pm(Also yeah, @mentions only works if you have 20 posts, same for voting in forum polls and rating posts. Using the embedded shoutbox or sending PMs requires 5, though)
Guess I should start posting more! ;)



Latest update:
  • [3/18/2016 8:42 PM] Migration will occur in less than 20 minutes! Please backup ops for your channel, and get ready for the switch! Remember, the service will be down for around 3 hours, or until things are completely set up!

DJ Omnimaga

In my case, it's not that $12/mo (which is more like $17/mo in Canadian dollars) is unsustainable for me, since I have a full time job, but in USA university and college tuition fees are 20-100 times higher than in Canada sometimes, which ruins many people, and I often hear that even in rural areas an appartment is very expensive compared to over here. >.<

In our cases, we used the basic services with a $10/mo plan (1 GB RAM, single-core, 30 GB SSD and 2 TB transfer, plus the weekly backup service that is $2 a month that once made things much easier to restore the site from a major incident. Back then, none of the staff could perform backups off-site to their computer at home or something, so we had to rely on DigitalOcean's. But now Streetwalrus performs daily backups on his server at home so if, for example, aeTIos decided to do the same, then we could probably do away with that paid service. Besides that we are not using any other special services I think. I also forgot if this is a managed or unmanaged VPS, since I don't run it myself.

Juju

Yeah, the $10/month plan is enough for what we need, nginx/PHP, email and a few IRC bots. Unless we constantly withstand a Slashdot effect, I think we'll be okay.
Read Zarmina!
YUKI-CHAAAANNNN
In the beginning there was walrii. In the end there will be walrii. All hail our supreme leader :walrii: --Snektron



if you wanna throw money at me and/or CodeWalrus monthly it's here

DJ Omnimaga

I'm actually curious about how well the server would do against a Slashdot effect, such as if a notable calc game was posted on Reddit...

Streetwalrus

Quote from: alberthrocks on March 18, 2016, 05:26:34 pm
@Streetwalrus: use the main server form and state that you don't want to use the services anymore!

Alright, done.

alberthrocks

March 19, 2016, 10:16:21 pm #12 Last Edit: March 19, 2016, 10:24:16 pm by alberthrocks
withgusto ZNC Migration Complete!
Thanks to everyone for their patience. We have now successfully migrated from our old server to this shiny new one!


  • You can still connect with the old server name, without any configuration changes. You can use the alternative server name if you are having issues.

  • The web interface is now only accessible via the new URL (sent via email). No more ports needed! This should help with any firewalls preventing you from getting out.

  • We have changed the SSL certificate from withg.org's ComodoSSL to the new Let's Encrypt! The old SSL certificates tend to cost around $10/year... why pay that when you can get it for free? :D

    • There's a custom renewal system in place for Let's Encrypt - the regular one is a bit memory heavy daemon-wise, and we only have 128 MBs of RAM on this server!

    • We decided to use https://github.com/diafygi/acme-tiny in place of that, along with a custom setup to update the certs for both our web server and ZNC. This is something I've been working on as an experiment on a cheap, smaller VPS, and so far it has been working well on that one - Let's Encrypt on a 64 MB VPS!

    • That said, despite testing, things can always still break in the future - please let us know if you have any problems with connecting to ZNC!


  • In addition, we have upgraded all of our SSL to super strength (A+ on the SSL test)! We figured that if we have time to migrate, we should have time to do security right!

  • Beyond that, we also enabled Cloudflare SSL. However, with the free Cloudflare SSL, this comes at a cost - IE on Windows XP will no longer be supported. Specifically, SNI support is not available at all on Windows XP via IE. However, if you are on Windows XP, using Firefox or Chrome will still work. Those browsers do not use Windows XP APIs, and bundle their own support for SNI. To say that again - if you are on Windows XP, just use Firefox/Chrome and things will work! If you must use IE on Windows XP for whatever reason, please let me know. If there is a serious need, I may be able to set something up for you!

  • Internally, logs have been converted to the new directory-based structure. It used to be all stored in one directory, resulting in thousands and thousands of files (and slowness)! For currently active users, this has been reorganized - you do not have to do anything on your part. In the future, we may be using this new structure for some potential new ZNC features down the road... stay tuned!

  • This time around, we're hoping for much better uptime! From initial impressions, ZNC seems to be faster at connecting to IRC, and most (if not all) of the BindHosts are resolving correctly. In addition, since this server is separate from the main one, uptime should hopefully be better - less impact from other people on the server! (In fact, the only person on this server is me!)

  • Enough technical stuff. Finally, we have some awesome new hostnames available for the picking! (Totally not inspired from rcfreak0! :P)

    Here is the list:
    znc.withg.org
    team.windows.znc.withg.org
    team.linux.znc.withg.org
    team.mac.znc.withg.org
    fancy.schmancy.znc.withg.org
    powered.by.the.amazing.znc.withg.org
    dont.look.behind.this.znc.withg.org
    food.for.thought.znc.withg.org
    yum.znc.withg.org
    sparkles.znc.withg.org


    Under your user settings, simply change your BindHost to your favorite, then click Save at the bottom. You can also change this per network if you want different hostnames on different networks.

  • znc-push has also been installed. If you have PushBullet (or some other push messaging app), and you want notifications whenever someone on IRC highlights you or PMs you, this is for you! (It basically allows you to get important notifications from IRC on your phone, desktop, etc.)

  • That's it! If you haven't already, and you are a ZNC user, please fill this out: http://goo.gl/forms/VFvTjFVpke






Quote from: DJ Omnimaga on March 19, 2016, 06:13:46 am
In my case, it's not that $12/mo (which is more like $17/mo in Canadian dollars) is unsustainable for me, since I have a full time job, but in USA university and college tuition fees are 20-100 times higher than in Canada sometimes, which ruins many people, and I often hear that even in rural areas an appartment is very expensive compared to over here. >.<

In our cases, we used the basic services with a $10/mo plan (1 GB RAM, single-core, 30 GB SSD and 2 TB transfer, plus the weekly backup service that is $2 a month that once made things much easier to restore the site from a major incident. Back then, none of the staff could perform backups off-site to their computer at home or something, so we had to rely on DigitalOcean's. But now Streetwalrus performs daily backups on his server at home so if, for example, aeTIos decided to do the same, then we could probably do away with that paid service. Besides that we are not using any other special services I think. I also forgot if this is a managed or unmanaged VPS, since I don't run it myself.
Ah, DigitalOcean - that's one of the options that I may switch to, or potentially a 512 MB server from them. I was thinking Linode, since the prices are similar and I know someone who owns one!

I'm not sure if the site has any sort of Cloudflare protection on it, but if a Slashdot thing ever occurred, you could potentially use their caching service to keep things sane. Maybe some server side caching too... but without a doubt, I think you guys will be fine!

@Streetwalrus: Just saw, thanks! :D

DJ Omnimaga

At least it wasn't postponed again like the other time :P

Also, interesting host names you got there. :P

alberthrocks

@Juju @DJ Omnimaga Do you have a DigitalOcean code that I can use? I want the $10 credit, and I'd prefer if the referral went to someone I know! :D

Powered by EzPortal