Join us on Discord!
You can help CodeWalrus stay online by donating here.

Scaling algorithms

Started by gameblabla, September 09, 2016, 02:05:54 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

gameblabla

Everyone knows scaling algorithms such as linear, cubic or lanczos or even specific ones like hqx or xBRZ.
But i thought i would talk about some of the lesser known scaling algorithms i think are pretty interesting.

Let's start with Waifu2x first.
Waifu2x is an upscaling algorithm for anime-style art that was developped at a chinese university in Hong Kong, inspired by SRCNN.
(That's probably the only good thing China has done, read on)
It also supports noise cancellation but a picture with little to no noise gives me the best result.
The goal of the algorithm is to give sharper details to the art, compared to traditional algorithms.
Perhaps the best thing to explain it is to show it.


Left is cubic upscaling, Right is Waifu2x

As you can see, the Waifu2x processed image has much less blur and has sharper edges than the cubic one.
It is overall much prettier to look at than the cubic processed one.

Another test, here's the reference image. (it comes from School Days)


What i have done next is to upscale it using the cubic algorithm to 1920x1080.
Then, i tried to upscale it twice (since waifu2x only upscales two times) with waifu2x.

Here's the result (You need Firefox or Webkit to view the APNG picture)
https://gameblabla.nl/img/waifu2x_result.png

This time though, it is not as significant as the previous one, probably due to many details in the background.
But the edges around the boy are still much sharper than the cubic one, once again and the overall picture is less blurry.

I tried with other anime pictures and i got pretty good results, so far i am very happy.
Maybe i should try to upscale an old anime to 2k using waifu2x ?
Sadly, i do not have the processing power or space for that.

Moving on to...

Seam Carving
This algorithm has a completely different purpose.
It was made in 2005 by a team in MERL with the goal of making an upscaler/downscaler that would keep most of details
in the picture when changing aspect ratios.

ImageMagick has an implementation of Seam Carving so i thought trying it out.
Our reference image is still the same :


I will reduce to 75% of its horizontal size. (out of 100%)


Here's what it would like when downscaled with Lanczos


You can see how it tries to keep important details like the main character without changing the proportions too much.
It results in a funny and non-consistent look though, especially for humans.
Let's try it with Obama instead :

Original:


Seam carved to 70% of its horizontal size :


Our president looks much thinner, he almost looks like a kid now !

Now let's try to downscale using this algorihm with our School Days picture :


Everything we knew about the picture before is now lost, it looks like something right from a nightmare...
Overall, it is best suited for paysages and aspect ratio changes.

Scalable Function Graphics
If you are looking for the best upscaling algorithm for image recovery, this is the one.
Designed for photos, the results are simply stunning :
The picture is much less blurry than any other algorithms while managing to be fairly accurate to
the original picture.
Have a look :


Photofunction, the only software based around it, can upscale such an image using the SFG algorithm.
However this comes at a cost : on a AMD FX chip, upscaling a 64x32 picture can take up to 12 hours !
It is thus not very practical but it does not take advantage of graphics cards, which could help.
It is also crap for aurora beams. (where a simple algorithm such as cubic fairs better)

It makes me wish someone else would reimplement it using CUDA or OpenCL because the results are great.
Sadly, due to overheating, i can't demonstrate it to you.
But you can download Photofunction here (requires Java 8.0) :
https://gameblabla.nl/files/photofunction.zip
  • Calculators owned: None (used to own an Nspire and TI-89)

Yuki

I heard wonders about Waifu2x, works best with vector art, like anime and cartoon art. (Which is pretty much made for.)

Seam Carving looks pretty interesting too, I think I heard of it while reading the ImageMagick docs.

And Scalable Function Graphics looks pretty interesting too, wish someone makes an implementation in CUDA or something like they did with Waifu2x.
  • Calculators owned: TI-83+ (dead?), Casio Prizm (also dead???)
  • Consoles, mobile devices and vintage computers owned: A lot
Read Zarmina!
YUKI-CHAAAANNNN
In the beginning there was walrii. In the end there will be walrii. All hail our supreme leader :walrii: --Snektron

if you wanna throw money at me and/or CodeWalrus monthly it's here

Dream of Omnimaga

Woah, I didn't know about those first two algorithms. The first one, Waifu2x, looks especially nice. Something I always wondered, though, is if there is a tool that can bulk resize an entire sprite sheet or a bunch of images rather than one by one. That could be handy for pixel art resizing because resizing an entire sprite sheet causes each sprite to bleed on the corners of the one next to it.
  • Calculators owned: TI-82 Advanced Edition Python TI-84+ TI-84+CSE TI-84+CE TI-84+CEP TI-86 TI-89T cfx-9940GT fx-7400G+ fx 1.0+ fx-9750G+ fx-9860G fx-CG10 HP 49g+ HP 39g+ HP 39gs (bricked) HP 39gII HP Prime G1 HP Prime G2 Sharp EL-9600C
  • Consoles, mobile devices and vintage computers owned: Huawei P30 Lite, Moto G 5G, Nintendo 64 (broken), Playstation, Wii U

catastropher

We had to implement seam carving in my algorithms class (which I took like 3 years ago). It's a fun dynamic programming problem! :D
  • Calculators owned: TI-83+, TI-83+ SE, TI-84+ SE, TI-Nspire CX, TI-92+, TI-89 Titanium
Creator of X3D, a 3D portal rendering game engine for Nspire, 68k, and PC

Dream of Omnimaga

I don't want to see what :walrii: would look like if seam carving was applied to it O.O
  • Calculators owned: TI-82 Advanced Edition Python TI-84+ TI-84+CSE TI-84+CE TI-84+CEP TI-86 TI-89T cfx-9940GT fx-7400G+ fx 1.0+ fx-9750G+ fx-9860G fx-CG10 HP 49g+ HP 39g+ HP 39gs (bricked) HP 39gII HP Prime G1 HP Prime G2 Sharp EL-9600C
  • Consoles, mobile devices and vintage computers owned: Huawei P30 Lite, Moto G 5G, Nintendo 64 (broken), Playstation, Wii U

catastropher

Quote from: DJ Omnimaga on September 25, 2016, 05:06:50 PM
I don't want to see what :walrii: would look like if seam carving was applied to it O.O
I still have my source code, maybe I should run Walrii through it? :P
  • Calculators owned: TI-83+, TI-83+ SE, TI-84+ SE, TI-Nspire CX, TI-92+, TI-89 Titanium
Creator of X3D, a 3D portal rendering game engine for Nspire, 68k, and PC

Dream of Omnimaga

Oh noes D:. But yeah this algorithm probably works best with people photos with the right coloring/gamma/contrast, right?
  • Calculators owned: TI-82 Advanced Edition Python TI-84+ TI-84+CSE TI-84+CE TI-84+CEP TI-86 TI-89T cfx-9940GT fx-7400G+ fx 1.0+ fx-9750G+ fx-9860G fx-CG10 HP 49g+ HP 39g+ HP 39gs (bricked) HP 39gII HP Prime G1 HP Prime G2 Sharp EL-9600C
  • Consoles, mobile devices and vintage computers owned: Huawei P30 Lite, Moto G 5G, Nintendo 64 (broken), Playstation, Wii U

gameblabla

Quote from: DJ Omnimaga on September 25, 2016, 04:58:50 PM
Woah, I didn't know about those first two algorithms. The first one, Waifu2x, looks especially nice. Something I always wondered, though, is if there is a tool that can bulk resize an entire sprite sheet or a bunch of images rather than one by one. That could be handy for pixel art resizing because resizing an entire sprite sheet causes each sprite to bleed on the corners of the one next to it.
All the waifu2x converters such as waifu2x-converter-cpp are command-line so yes, you can convert multiple sprites by simply doing a loop
like this (bash) :
for f in $(find . -type f); do
./waifu2x-converter-cpp -i $f;
done

And yes, it could be pretty hand for a video game if you want to work on a higher-resolution remake or something.
Or you suck badly at drawing high-resolution graphics.

QuoteI still have my source code, maybe I should run Walrii through it? :P
I would like to see it, especially if it's written in C/C++ !
It could be implemented to an emulator.

QuoteOh noes D:. But yeah this algorithm probably works best with people photos with the right coloring/gamma/contrast, right?
It works only best without any humans in it or else they will be badly deformed.
  • Calculators owned: None (used to own an Nspire and TI-89)

Dream of Omnimaga

Yeah my main goal at one point was to port Reuben to PC or higher-res, but I didn't want to redo all sprites from scratch. :P (except the main character, which would have used the battle sprite facing all 4 directions)
  • Calculators owned: TI-82 Advanced Edition Python TI-84+ TI-84+CSE TI-84+CE TI-84+CEP TI-86 TI-89T cfx-9940GT fx-7400G+ fx 1.0+ fx-9750G+ fx-9860G fx-CG10 HP 49g+ HP 39g+ HP 39gs (bricked) HP 39gII HP Prime G1 HP Prime G2 Sharp EL-9600C
  • Consoles, mobile devices and vintage computers owned: Huawei P30 Lite, Moto G 5G, Nintendo 64 (broken), Playstation, Wii U

VMike

Hello.
First I want to say sorry that I dug up this old thread. I just accidently stumbled on it in my persue for info.
So I registered here to write a reply in this thread.
It is about Waifu2x that I think makes old games look soooo good. Here arw some pictures of Waifu2x 32x with highest noise reduction over Final Fantasy VII. It is first rendered in the standard resolution of 320x240 unstretched wich gives Waifu2x best results.

https://m.imgur.com/a/Zef82
And also here
https://m.imgur.com/a/LOkFG

Now, I know there are other filters for Ps1 emulation but none I have ever seen gives such nice result.
I have asked around everywhere but always get thevsame answer; No, Waifu2x is not possible to run in realtime.
And I have kinda given up. But since I got nothing to lose I tought I might as well try here. If anyone is up for the task, I would even pay a little to have Waifu2x filtering my Ps1 emulators in real time.
Oh well. Thanx for reading anyway :)

Dream of Omnimaga

Hi @VMike and welcome to the forums. To be honest, I am not too sure if I like how the Final Fantasy VII screenshots turned out with Waifu2x, as there seems to be too much filtering (probably due to the original 240p resolution being too low), but it actually gives the game a different feel, like if it was painted by hand. I am definitively curious if computers are powerful enough to render such filter in real time. I wonder if it would be possible to add less filtering, though, so that the game looks closer to PS1 graphics but at higher resolution? What about 480p?
  • Calculators owned: TI-82 Advanced Edition Python TI-84+ TI-84+CSE TI-84+CE TI-84+CEP TI-86 TI-89T cfx-9940GT fx-7400G+ fx 1.0+ fx-9750G+ fx-9860G fx-CG10 HP 49g+ HP 39g+ HP 39gs (bricked) HP 39gII HP Prime G1 HP Prime G2 Sharp EL-9600C
  • Consoles, mobile devices and vintage computers owned: Huawei P30 Lite, Moto G 5G, Nintendo 64 (broken), Playstation, Wii U

VMike

#11
Thank you for your answer.
In those pictures I have taken an image from the emulator while it runs in 320x240. Its wierd, because when it is pre-rendered games like Final Fantasy and Resident Evil they actualy looks better if the source is in original resolution. I have contacted a guy on twitter as well who claims that it should be easy to make Waifu2x run in 30fps if the image is small enough. I hope 320x240 is small enough.

Could someone make like an overlay you just could add over an emulator window wich uses Waifu2x? Its wierd, because everywhere else I have asked they have said that it is impossible to make it run fast enough for Ps1 emulation.

Yeah I really, really like that look it gives the game. Would be extremely nice if someone could make it work.

Here are some pictures of Resident Evil 2 320x240 screenshots upscaled with Waifu2x.




I also added some minor ReShade to them. Gaussian flare and some sharpening. Well, compared to the originals.. this looks like gold. At least to me :)

c4ooo

What does a waifu2x walrus look like? :3

gameblabla

#13
Hey @VMike,
as you may have noticed when using Waifu2x, it is very CPU-intensive !
My AMD FX 6300 can take up to 2 minutes to upscale to a 2k image, let alone for 4k...
For real-time upscaling, it needs to be much faster than that, at least 30 images per second.

The only way for it to be practical way would be to make use of GPU acceleration (via CUDA or OpenCL) but even then,
it would require a powerful GPU (at least a Titan), fast storage (Samsung SSD), fast DDR4 memory etc...
Not impossible but very impractical for most people.

Also, Waifu2x would be best suited for visual games and games with anime-styled graphics.
Resident Evil looks interesting with Waifu2x, almost like a painting.
But for now, it's best suited for still pictures and maybe movies (if you are doing frame-by-frame).
  • Calculators owned: None (used to own an Nspire and TI-89)

VMike

Quote from: gameblabla on March 29, 2017, 06:37:02 AM
Hey @VMike,
as you may have noticed when using Waifu2x, it is very CPU-intensive !
My AMD FX 6300 can take up to 2 minutes to upscale to a 2k image, let alone for 4k...
For real-time upscaling, it needs to be much faster than that, at least 30 images per second.

The only way for it to be practical way would be to make use of GPU acceleration (via CUDA or OpenCL) but even then,
it would require a powerful GPU (at least a Titan), fast storage (Samsung SSD), fast DDR4 memory etc...
Not impossible but very impractical for most people.

Also, Waifu2x would be best suited for visual games and games with anime-styled graphics.
Resident Evil looks interesting with Waifu2x, almost like a painting.
But for now, it's best suited for still pictures and maybe movies (if you are doing frame-by-frame).

There is a version using cuda. I think that is also open sourse. Yes it is slow, with my 980ti it takes about 20 seconds to scale 320x240 32 times the size. It was just that I read that someone claims he has it got running for 30 fps. And I dont think it is impossible.

I really want this for those old pre-rendered psone games. I have not seen any other filter that makes them look so good.

Powered by EzPortal