The magical charm of Studio Ghibli’s art style has captivated fans for decades, making it a dream for many to transform their own photos into this hand-painted aesthetic. While some AI-powered tools offer Ghibli-style conversions, they often come with restrictions, either locked behind paywalls or requiring processing that compromises privacy. Many people are also raising concerns about privacy and data theft associated with sending personal and sensitive pictures to some AI company’s server. But what if you could convert your pictures to stunning Ghibli-style images entirely on your own device, with full control over the process? Thanks to EasyControl, an exciting open-source model, you can now achieve exactly that, privately, locally, and for free without any limits!
In this guide, we’ll walk you through a step-by-step process to set up and run this powerful AI model on your own system. By the end, you’ll be able to generate breathtaking Ghibli-inspired portraits without relying on external services, all while keeping your data private and your creativity limitless.
Prerequisites
The minimum system requirements for running this model are:
Step-by-step process to install and run EasyControl locally
For the purpose of this tutorial, we’ll use a GPU-powered Virtual Machine by NodeShift since it provides high compute Virtual Machines at a very affordable cost on a scale that meets GDPR, SOC2, and ISO27001 requirements. Also, it offers an intuitive and user-friendly interface, making it easier for beginners to get started with Cloud deployments. However, feel free to use any cloud provider of your choice and follow the same steps for the rest of the tutorial.
Step 1: Setting up a NodeShift Account
Visit app.nodeshift.com and create an account by filling in basic details, or continue signing up with your Google/GitHub account.
If you already have an account, login straight to your dashboard.
Step 2: Create a GPU Node
After accessing your account, you should see a dashboard (see image), now:
- Navigate to the menu on the left side.
- Click on the GPU Nodes option.
- Click on Start to start creating your very first GPU node.
These GPU nodes are GPU-powered virtual machines by NodeShift. These nodes are highly customizable and let you control different environmental configurations for GPUs ranging from H100s to A100s, CPUs, RAM, and storage, according to your needs.
Step 3: Selecting configuration for GPU (model, region, storage)
- For this tutorial, we’ll be using 1x RTX A6000 GPU, however, you can choose any GPU as per the prerequisites.
- Similarly, we’ll opt for 200GB storage by sliding the bar. You can also select the region where you want your GPU to reside from the available ones.
Step 4: Choose GPU Configuration and Authentication method
- After selecting your required configuration options, you’ll see the available GPU nodes in your region and according to (or very close to) your configuration. In our case, we’ll choose a 1x RTX A6000 48GB GPU node with 64vCPUs/63GB RAM/200GB SSD.
2. Next, you’ll need to select an authentication method. Two methods are available: Password and SSH Key. We recommend using SSH keys, as they are a more secure option. To create one, head over to our official documentation.
Step 5: Choose an Image
The final step is to choose an image for the VM, which in our case is Nvidia Cuda, where we’ll deploy and run the inference of our model.
That’s it! You are now ready to deploy the node. Finalize the configuration summary, and if it looks good, click Create to deploy the node.
Step 6: Connect to active Compute Node using SSH
- As soon as you create the node, it will be deployed in a few seconds or a minute. Once deployed, you will see a status Running in green, meaning that our Compute node is ready to use!
- Once your GPU shows this status, navigate to the three dots on the right, click on Connect with SSH, and copy the SSH details that appear.
As you copy the details, follow the below steps to connect to the running GPU VM via SSH:
- Open your terminal, paste the SSH command, and run it.
2. In some cases, your terminal may take your consent before connecting. Enter ‘yes’.
3. A prompt will request a password. Type the SSH password, and you should be connected.
Output:
Next, If you want to check the GPU details, run the following command in the terminal:
!nvidia-smi
Step 7: Set Up Project Environment
- Ensure you’ve have Anaconda installed and create virtual environment with
conda.
conda create -n ghibli python=3.11 -y && conda activate ghibli
Output:
2. Clone the official repository of EasyControl and move inside the project directory.
git clone https://github.com/Xiaojiu-z/EasyControl.git && cd EasyControl
Output:
3. Install project dependencies.
pip install -r requirements.txt
Output:
4. Login to huggingface-cli with HF READ token.
(replace <YOUR_HF_TOKEN> with your own access token)
huggingface-cli login --token=<YOUR_HF_TOKEN>
Output:
Step 8: Download model files
- Next, we’ll download the model from HuggingFace using the command below:
huggingface-cli download --resume-download Xiaojiu-Z/EasyControl --local-dir checkpoints --local-dir-use-symlinks False
Output:
2. Open the project in VSCode Editor.
For this, if you’re using a remote server (e.g. NodeShift GPU), you’ll first need to connect your local VS Code editor to your remote server via SSH with the following steps:
a) Install the “Remote-SSH” Extension by Microsoft on VS Code.
b) Type “Remote-SSH: Connect to Host” on the Command Palette.
c) Enter the host details, such as username and SSH password, and you should be connected.
3. Once inside the project, replace the code in app.py with the below code snippet.
import spaces
import os
import json
import time
import torch
from PIL import Image
from tqdm import tqdm
import gradio as gr
from safetensors.torch import save_file
from src.pipeline import FluxPipeline
from src.transformer_flux import FluxTransformer2DModel
from src.lora_helper import set_single_lora, set_multi_lora, unset_lora
# Initialize the image processor
base_path = "black-forest-labs/FLUX.1-dev"
lora_base_path = "./checkpoints/models"
pipe = FluxPipeline.from_pretrained(base_path, torch_dtype=torch.bfloat16)
transformer = FluxTransformer2DModel.from_pretrained(base_path, subfolder="transformer", torch_dtype=torch.bfloat16)
pipe.transformer = transformer
pipe.to("cuda")
def clear_cache(transformer):
for name, attn_processor in transformer.attn_processors.items():
attn_processor.bank_kv.clear()
# Define the Gradio interface
@spaces.GPU()
def single_condition_generate_image(prompt, spatial_img, height, width, seed, control_type):
# Set the control type
if control_type == "Ghibli":
lora_path = os.path.join(lora_base_path, "Ghibli.safetensors")
set_single_lora(pipe.transformer, lora_path, lora_weights=[1], cond_size=512)
# Process the image
spatial_imgs = [spatial_img] if spatial_img else []
image = pipe(
prompt,
height=int(height),
width=int(width),
guidance_scale=3.5,
num_inference_steps=25,
max_sequence_length=512,
generator=torch.Generator("cpu").manual_seed(seed),
subject_images=[],
spatial_images=spatial_imgs,
cond_size=512,
).images[0]
clear_cache(pipe.transformer)
return image
# Define the Gradio interface components
control_types = ["Ghibli"]
# Create the Gradio Blocks interface
with gr.Blocks() as demo:
gr.Markdown("# Ghibli Studio Control Image Generation with EasyControl")
gr.Markdown("The model is trained on **only 100 real Asian faces** paired with **GPT-4o-generated Ghibli-style counterparts**, and it preserves facial features while applying the iconic anime aesthetic.")
gr.Markdown("Generate images using EasyControl with Ghibli control LoRAs.（Due to hardware constraints, only low-resolution images can be generated. For high-resolution (1024+), please set up your own environment.）")
gr.Markdown("**[Attention!!]**：The recommended prompts for using Ghibli Control LoRA should include the trigger words: `Ghibli Studio style, Charming hand-drawn anime-style illustration`")
gr.Markdown("😊😊If you like this demo, please give us a star (github: [EasyControl](https://github.com/Xiaojiu-z/EasyControl))")
with gr.Tab("Ghibli Condition Generation"):
with gr.Row():
with gr.Column():
prompt = gr.Textbox(label="Prompt", value="Ghibli Studio style, Charming hand-drawn anime-style illustration")
spatial_img = gr.Image(label="Ghibli Image", type="pil") # 上传图像文件
height = gr.Slider(minimum=256, maximum=1024, step=64, label="Height", value=768)
width = gr.Slider(minimum=256, maximum=1024, step=64, label="Width", value=768)
seed = gr.Number(label="Seed", value=42)
control_type = gr.Dropdown(choices=control_types, label="Control Type")
single_generate_btn = gr.Button("Generate Image")
with gr.Column():
single_output_image = gr.Image(label="Generated Image")
# Link the buttons to the functions
single_generate_btn.click(
single_condition_generate_image,
inputs=[prompt, spatial_img, height, width, seed, control_type],
outputs=single_output_image
)
# Launch the Gradio app
demo.queue().launch()
The structure looks like this in the editor:
Step 9: Run and Access the Model with Gradio Interface
- Finally, come back to the terminal and run the following command which will load the model checkpoints and launch the app with Gradio.
python app.py
Output:
Sometimes, you might get an error of a missing dependency named “spaces”. Just run the following command to install that:
pip install spaces
After that run the previous command again and it should start the process.
2. Once the app is up and running, forward and tunnel SSH port to access the app in your local browser, if you’re using a remote server (e.g. NodeShift).
Run the following command in your local terminal after replacing:
<YOUR_SERVER_PORT> with the PORT allotted to your remote server (For the NodeShift server – you can find it in the deployed GPU details on the dashboard).
<PATH_TO_SSH_KEY> with the path to the location where your SSH key is stored.
<YOUR_SERVER_IP> with the IP address of your remote server.
ssh -L 7860:localhost:7860 -p <YOUR_SERVER_PORT> -i <PATH_TO_SSH_KEY> root@<YOUR_SERVER_IP>
Output:
3. Access the app at http://localhost:7860.
It looks like this:
Step 10: Model Test Examples
Below we have attached some of the examples that we obtained while testing this model:
Note: To trigger the model for the best output, your prompt must contain these words:
Ghibli Studio style, Charming hand-drawn anime-style illustration
Example-1
Output Image:
Example-2
Output Image:
Example-3
Output Image:
Conclusion
Transforming your photos into enchanting Studio Ghibli-style artwork is now achievable with the EasyControl model, offering a local and private solution that ensures your data remains secure. However, generating creative images and art like these demands sufficient compute power. Therefore, for users seeking enhanced computational power without compromising privacy, NodeShift’s decentralized cloud platform provides scalable infrastructure and optimized deployment capabilities, making it easier to manage AI-driven environments securely.
