I shouldn’t be doing what I am doing as I evidently do not know what I am doing. And yet here I am.
So maybe one of you knows more than the rest of the internet.
My plan is to have the following setup working
windows laptop:
WSL2:
ansible controller (or whatsitcalled) inside WSL2
vagrant creating VMs on windows with virtualbox as provider
vagrant installed and nominally working in WSL2
vagrant actually creating VMs from WSL2 … NOPE
later…:
testing my ansible playbooks via vms created with vagrant
deploying my tested playbooks to my server
checking my configurations into git
(and that is without getting into WHAT I want to do on the server. Setting it all up more or less manually, I would probably be done already but that is not the point)
Vagrant does create VMs when I start it inside WSL2 but it hangs on testing ssh connection because it doesn’t have access to the network the VM lives in and I have now spent a whole day trying to figure out a way around this. Running the same vagrantfile in powershell gives me a VM I can log into.
So my best guess is that I need to improve my networking configuration on WSL2.
(The other way round works fine. F.e. for work I am running docker inside WSL2 and I have a powershell script that does port-forwarding from WSL2 to windows. But I cannot get the other way to work)
Does anyone have any expertise in this? Am I even asking the right questions? I have already exhausted my local network-able friends and stackoverflow and superuser have a lot of posts on this but most people cannot even get Virtualbox and WSL2 running at the same time which is not my problem.
I will also take any other setup that allows me to test ansible playbooks before I use them on my live server. It does need to be something though that works from commandline. If necessary even powershell (I hate powershell)… but ansible generally doesn’t like windows. It’s a surprise it deigns to work inside WSL.
Docker I’ve got already–thanks to needing it for work
The problem is mostly that networking vs me is a fight I am losing…
Especially with all the shenanigans with host-only vs bridged vs nat what Virtualbox is doing and with WSL2 being a VM itself…
I have almost resigned myself to compromising with the setup. I imagine I could setup the vagrant machine from powershell which works and then I only need to somehow make it so WSL can see that machine manually and instead of using vagrant to provision with ansible directly I will manually trigger ansible to run against the VM.
This leaves me with enough pieces to figure out the full process later and still give me a way to test before deploying.
I even made diagrams of what I want it to look like on the server…
docker container for my 2 wordpress blogs
docker container for my mail server setup with postfix/dovecot/roundcube
docker container next cloud
docker container for my own homebrew tools and static webcontent
all nicely hidden behind firewall and with an nginx to distribute everything–a bit of basic port forwarding isn’t too complicated for me. The most complicated part of the main server is going to figure out letsencrypt. I’ve used plesk for this up to now and even that created some problems
If I get this configured into an ansible playbook, I have configurations that I can commit to my git instead of keeping a running tally of what I did in some kind of text document.
The only thing I haven’t quite figured out is how to do a clean separation and backup of my data so it doesn’t get mixed up with the configurations.
The thing is that part is fun because I have done most of these things before successfully. And by the time I am there I am working with linux and not windows and there will be some progress… stuff I can look at in a browser and see my work getting done. Getting the tools ready is not visible and sooo frustrating. The worst part of my current job was setting up my workspace.
Oh foobar, this has been ages. But with job and renovations somehow this never got done.
So I am somewhat back at it because my nextcloud is being facetious. Mail settings are also deteriorating although I am unsure that it is really me or just some big a…hole providers blocking off small mailservers as “not trustworthy” (looking at you gmail)
So I think my biggest hurdle… seems to be domains. I want to be able to test the new server while leaving the old one up and running. So what I need is some kind of temporary domain (I have enough domains to set up something) which I then want to switch back to the one that is in use now. Is there a way to do this somewhat safely (meaning I don’t want to have hours of downtime because of stupid editing mistakes during the switchover)?
This is not a question about DNS, my DNS provider has a nice web interface that allows me to change around where my Domains and or subdomains or services are pointing at any time.
This is purely how to minimize impact while I am switching over from test to production mode.
In my mind I am doing most domain configurations for anything web in nginx and then forward to the docker container in question.
But what about the mailserver? This is also the most sensitive to lengthy downtimes… everything else is really quite optional.
I have given up on doing anything fancy with ansible et al. I would be happy if I just got a basic server up and running without plesk.
Data management and migration will be the biggest hurdle I think. Are you building this on a greenfield server? Can you map the old data to the new server over NFS or CIFS? What kind of backend is the data being stored on now?
Do you already have some of the functionality up and running in Docker? I think getting familiarity with Docker is the first step which can be done with sample data.
I use docker for work … sure only one specific setup but I think I am fine with setting it up and containers and mounting in directories for the data etc. much more comfortable than I am with ansible. I previously used docker for my server before switching over to the virtual one and using plesk.
Migration is the most difficult part definitely. some of the tools I use should be easily migrated via db dumps and scp …
I have previously migrated mailserver, wordpress. Nextcloud I am not sure… most of the non-file based stuff is in the database. the files I could probably reupload or scp. It think that one just uses a directory to store the files.
The impact I most expect is that I run everything on a test domain, then migrate my data and point my main domain + subdomains at the new server and some internal configuration is forgotten and the thing instead of “mydomain.de” thinks it is called “mytestdomain.de” and foobars everything and debugging takes forever. Ideally the services are agnostic to what domain they are running on but in my experience they are not. (Wordpress is definitely guilt of this stuff, mailserver is the thing I am most afraid of)
Backend… file-system and mysql … I have some of the files on an “external” virtual drive that I might be able to remount to the new server. I am not sure. This is all virtual server stuff. I have a hard time explaining these things. Sorry -.-
Ah Grüne Wiese. Yes yes, the new server is empty for now.
Well, the great thing about (well-designed) docker containers is that they are stateless and they only have precepts about the environment based on what you configure on the container itself. Unfortunately, a problem that stems from that is that you can configure configuration and data mounts that can contain previously configured information and data.
I think I would start with doing some dry-runs with snapshot data following official or community migration guides, and see what kind of artifacts you encounter after fake migrations.
you’re right, I should just jump and get familiar with it all in some dry-runs and only think about migrating when I feel like I control the thing. I can reset the server as much as I like
Specific thing with DNS and mail: turn down the DNS TTL. You need to do this at least (TTL) before it matters. Then remote servers will update more often and spot changes sooner. (Except Microsoft ones, which deliberately break the standards and hold onto cached DNS data forever, perhaps because their own DNS servers are so ghastly.)
Also, set up a secondary MX. Then if mail isn’t being accepted by the primary for some reason it’ll pile up there without you needing to do anything. Doesn’t help if the primary is saying “no, I’m not that domain’sserver” though; for that you need to change DNS.
Right now I need to learn or rather relearn the reverse proxy setup I need. I think the last time I ran this I was doing that with apache. But these days I think one would do it in nginx. But I am getting there. I just pointed a few subdomains at my server and I setup nginx and letsencrypt and docker/docker-compose so once I figure out a little better how to play with nginx without plesk I think I am good to go with the docker side.
Thanks @pillbox I really needed someone to tell me to just start playing.
I am keeping a protocol but I know from past attempts these are faulty and end up having holes. Which is–I know I know–where ansible would come in. And maybe I will end up using that at some point. For now I just want to see some services running…
Being a lazy dev, I should know this right? Ansible seems like a huge problem when you start with it, I suspect it looks like a smaller problem when I know where I am going.
That looks pretty neat. I think for this run I will stick with nginx as I have already started that.
But as I am reasonably sure that I want to reset the server at least once to make sure I am eliminating mistakes and failed attempts I made along the way… maybe I will try that Automatic detection sounds lazy
Traefik looks very promising but I have yet to figure out https
I have managed to get it to route to a wordpress container.
The documentation leaves something to be desired it is very much the kind of tutorial that was written by someone who has no idea what people don’t know when they first start using the tool. but i found their forums (discourse what a surprise) and I’ll figure it out.
I could in theory configure nginx manually or a docker image specifically for using nginx as reverse proxy. but I doubt the nginxreverse proxy docker image is any easier to figure out.
One of the issues is that there are so many different ways to get the same result it is hard to find all the parts that belong to the same path.
This is the caveat I always mention when talking about the benefits of open source software. You have to curate an ecosystem; and that may take a lot of trial and error and possible major backtracking as you find the limits of the different pieces of software you are gluing together
I was trying to figure out how to update my PHP install the other day. Gave up in the end as all the answers were contradictory. Will try again another day.
I got as far as beginning to understand the mailserver container…
I remember why I hate setting up mailservers and why after doing so I immediately forget everything about the mailserver. It was totally the reason that plesk seemed/seems so attractive.
Who wants to know what amavis and clamd are?
When I read the “if you do this because you want to be able to use smtp from your other docker containers you may end up creating an open relay” I just want to give up already.
At least the documentation on this container is much much more thorough than everything else.
in theory I have all the containers that will give me the same functionality as my old server
but neither roundcube nor outlook want to connect to the mailserver something something ports… or SSL or who knows what with mailservers.
mailserver also needs a lot of tweaking to get our crazy mail setup right.
database initialization still requires me to go to adminer and input the sql script through the web console when it shouldn’t but maybe I just need to try again with the latest version and it might just work?
roundcube doesn’t want to use docker secrets for the database password oO
but most of it is there and requires very little manual fiddling now. and the great thing is the basic server installation remains very basic. all it needs is ssh key, ufw ports and installation of docker + docker-compose. ~20-30 commands.
Of course my docker-compose is about 400 lines long… but who cares about that. That’s gitted and automated.
