Using the Toonz Farm
The Toonz Farm allows you to process Tahoma scenes submitted as cleanup tasks or render tasks in the task list by using a series of computers connected on the same network using TCP/IP (see Cleaning up Drawings in Batch Mode and Rendering Scenes in Batch Mode ).
When tasks are executed, they are automatically assigned to the computers belonging to the farm in order to balance the workload. When all computers are busy, remaining tasks are not assigned until a machine is ready to process them.
Note
If you have divided a render task into chunks, each chunk will be assigned to a different computer of the farm, thus speeding up the rendering of a single render task (see Using Chunks when Rendering Tasks ).
The Toonz Farm is composed by the following:
Controller node: is the computer that dispatches the tasks to server (rendering) nodes. This computer runs the controller program, named
tfarmcontroller.exe
on Windows andtfarmcontrollerd
on Macintosh. One controller node is needed for defining a farm.
Note
The controller node can also be used as a server (rendering) node.
Server (rendering) nodes: are the computers that execute tasks dispatched by the controller. Each of these computers runs a server program, named
tfarmserver.exe
on Windows andtfarmserverd
on Macintosh, that is in charge of running tasks, notifying the controller when a task has been completed, etc.Root folder named FARMROOT: is a shared folder where a number of configuration files are stored. The controller node and all the rendering nodes must be able to access this shared folder (see Configuring the Toonz Farm ).
Client program: is the program run by the user to queue new tasks and to monitor the Toonz Farm. Tahoma itself is the client program of the Toonz Farm, and can be run on any computer on the network, not necessarily on a controller node or a sever (rendering) node.
Note
In order to render the scenes successfully all the computers involved in the Toonz Farm, controller, servers and clients have to share the same projectroot folders.
Installing the Toonz Farm on Windows
To install the Toonz Farm, you have to install the Farm Controller component on the controller node, and the Farm Server component on the server (rendering) nodes.
Note
If you want to use a computer both as controller and as server (rendering) node, you have to install both the components.
During the installation of any of the components you must specify the full-path of the FARMROOT folder. This path must be expressed using the UNC format, like the following:
\\HostName\ShareName\FolderName
The drive where the FARMROOT folder is located has to be shared on the network, giving full control to any user by setting the right permissions.
When you choose to install the Farm Controller component, the installation program will create for the controller program a Windows Service, named Toonz Farm Controller.
When you choose to install the Farm Server component, the installation program will create for the server program a Windows Service, named Toonz Farm Server.
Note
As the Toonz Farm requires exchanges of data between computers in the network by using TCP/IP protocol, any firewall system that may block access on computer ports has to be disabled.
Setting the Controller Node
On the controller node the controller component is installed as a Windows service, named Toonz Farm Controller. By default a Windows Service runs under the security credentials of the Local System account. By default this account is not granted the right to access folders shared by remote hosts.
Since the Toonz Farm Controller needs to access the FARMROOT folder, typically shared by a remote host, please run this service as a user with the proper rights.
Note
It is possible to start the service from a DOS shell by navigating to the folder where the binaries of Tahoma are installed, and then using the following command line:tfarmcontroller -console
Tip
To run the Toonz Farm Controller as a user with the proper rights:
Choose Control Panel → Administrative Tools → Services, and right-click the Toonz Farm Controller service to open the Properties panel.
Select the Log On page, check the This Account: option.
Choose the network where the Toonz Farm is going to work and set an account that has administrator rights on that network. Insert and confirm the related password.
Click the OK button.
Setting the Server Nodes
On the server (rendering) nodes, the server component is installed as a Windows service, named Toonz Farm Server. By default a Windows Service runs under the security credentials of the Local System account. By default this account is not granted the right to access folders shared by remote hosts.
Since the Toonz Farm Server, for rendering purposes, needs to access one or more folders shared by remote hosts, please run this service as an user with the proper rights.
Note
It is possible to start the service from a DOS shell by navigating to the folder where the binaries of Tahoma are installed, and then using the following command line:tfarmserver -console
Tip
To run the Toonz Farm Controller as an user with the proper rights:
Choose Control Panel → Administrative Tools → Services, and right-click the Toonz Farm Server service to open the Properties panel.
Select the Log On page, check the This Account: option.
Choose the network where the Toonz Farm is going to work and set an account that has administrator rights on that network. Insert and confirm the related password.
Click the OK button.
Installing the Toonz Farm on Macintosh
To install the Toonz Farm, you have to install the Farm Controller component on the controller node, and the Farm Server component on the server (rendering) nodes.
Note
If you want to use a computer both as controller and as server (rendering) node, you have to install both the components.
After the installation of the toonz farm components, a FARMROOT folder has to be defined, and on all the computers belonging to the farm, both controller and servers, the full-path of the FARMROOT folder has to be specified.
To define the FARMROOT folder you can take as model the toonzfarm
folder stored in Tahoma stuff
. You can take the toonzfarm
folder from any computer where Tahoma in installed, choose the location you prefer, and share it on the network, giving full control to any user by setting the right permissions.
To specify the path to the FARMROOT on the computers, both controller and servers, belonging to the farm, you have to edit the configfarmroot.txt
file by typing in it the full path to the FARMROOT folder.
To retrieve this file open the Applications/Tahoma
folder, right-click the Tahoma icon and choose Show Package Contents from the menu that opens; in the folder that opens browse to the Contents/Resources
folder and retrieve the configfarmroot.txt
file.
Note
As the Toonz farm requires exchanges of data between computers in the network by using TCP/IP protocol, any firewall system that may block access on computer ports has to be disabled.
Configuring the Toonz Farm
Once the installation has been completed, you have to configure the Toonz Farm by editing some configuration files stored in the FARMROOT folder.
The FARMROOT folder must be accessible by all the nodes that compose the Toonz Farm: the controller node, the server (rendering) nodes and the computers running Tahoma. This means that the drive where the FARMROOT folder is located has to be shared on the network, and the permissions for users have to be maximum, in order to give full control to any users.
For the same reason the computer where the FARMROOT is located has to be started before all the other computers involved in the farm.
The FARMROOT folder contains the following:
the
config
folder containing theservers.txt
, thecontroller.txt
and theid.txt
.the
logs
folder.
The controller.txt
and the servers.txt
configuration files must be edited to define the controller node and the server (rendering) nodes before using the Render Farm.
Defining the Controller Node
The controller node is defined in the controller.txt
configuration file placed in the FARMROOT\config
folder.
The file must contain a single text line that describes the node you want to use as a controller.
For the controller node you must specify the following:
Computer name.
IP address.
The port number you intend to use for the controller service.
Note
Be sure that the port you set is not blocked by any firewall system, and is open for all the computers belonging to the Toonz farm.
Each of the above information must be separated by a single space. Here is an example of a controller.txt
file:
render1 10.10.0.130 10000
In the above example, render1
is the controller node, its IP address is 10.10.0.166
and the controller program runs on the 10000
port.
Defining the Server Nodes
The server (rendering) nodes are defined in the servers.txt
configuration file placed in the FARMROOT\config
directory.
The file must contain a text line for every computer you want to use as a server (rendering) node.
For each sever (rendering) node you must specify the following:
Computer name.
IP address.
The port number you intend to use for the server service.
Note
Be sure that the port you set is not blocked by any firewall system, and is open for all the computers belonging to the Toonz Farm.
The above information must be separated by a single space. Here is an example of a servers.txt
file:
render1 10.10.0.166 8002
render2 10.10.0.195 8002
render3 10.10.0.111 8002
render4 10.10.0.180 8002
In the above example, render1
is a server node whose IP address is 10.10.0.166
and the server program runs on the 8002
port; render2
is a server node whose IP address is 10.10.0.195
and the server program runs on the 8002
port; etc.
If you want to add or remove a server (rendering) node from the servers.txt
file, you have to restart the controller node in order to make the changes effective.
Defining the Toonz Farm in Tahoma
Tahoma is the Toonz Farm client program: it allows you to submit new tasks and monitor the status of the farm.
If the computer where you run Tahoma is a controller or a server (rendering) node for the Toonz Farm, no further settings are required.
If the computer is neither a controller nor a server (rendering) node for the Toonz Farm, the FARMROOT has to be defined.
Tip
To define the FARMROOT in Tahoma:
Run Tahoma and go to the Farm room.
In the Batch Servers pane, set the Process with: option menu to Render Farm.
In the dialog that opens specify the full path to the FARMROOT folder.
Click the OK button.
Using the Toonz Farm
In Tahoma you can find the Farm room that allows you to monitor the farm processes. This room contains two main panels: the Tasks window and the Batch Servers window.
The render farm can be monitored by using both the Tasks pane, where the list of tasks can be managed and executed, and the Servers pane, where the computers executing the tasks can be managed.
Note
All shared disks involved in the task processing, i.e. disks where files resulting from the task execution have to be written, must be shared granting full permissions to any user, otherwise tasks execution will not succeed. For Windows, disks must have a Share Permission set to Full Control for Everyone; for Macintosh, the sharing settings must allow any user to read from and write to disks.
Monitoring the Server Nodes
The Batch Servers panel is divided in three areas: at the top there’s an option menu to set whether the tasks execution has to be performed with the Local computer or with the Render Farm; then there is the list of server (rendering) nodes in the farm; at the bottom you can see information about the server currently selected in the list.
The server (rendering) nodes included in the list are those defined in the server.txt
configuration file (see Defining the Server Nodes ). If a server node is down at the controller startup, it will not be available in the farm: this means that all the server nodes must be up before starting up the controller node. For the same reason if you want to restart or shut down a server (rendering) node, you have to restart the controller node in order to make the changes effective.
If the Process with: option menu is set to Local, all the tasks you run will be executed by the local computer; if it’s set to Render Farm, the tasks will be executed by the Toonz Farm.
Note
When you select Render Farm you could be prompted to define the FARMROOT (see Defining the Toonz Farm in Toonz ).
When a server is selected in the list, the following related information are displayed in the area at the bottom of the window:
Name: displays the name of the server (rendering) node.
IP Address: displays its IP address.
Port Number: displays the port number used to exchange data with the farm.
Tasks: displays the task being executed.
State: displays if the server is Ready, Busy or Down.
Note
A server node state may be down if the server port you set is blocked by a firewall system, and is not open for all the computers belonging to the Toonz Farm (see Defining the Server Nodes ).
Number of CPU: displays the number of CPUs available on the server.
Physical Memory: displays the amount of physical memory available on the server.
Tip
To use the Local computer to execute tasks:
Set the Process with: option menu to Local.
Tip
To use the Toonz Farm to execute tasks:
Set the Process with: option menu to Render Farm.
Checking the Toonz Farm Processing
When the Toonz Farm is used, any problem that may be experienced during the processing is recorded in LOG files, that can be opened and used to diagnose problems.
The LOG files are saved in the Tahoma stuff\toonzfarm
folder of the computer experiencing the problem; each LOG refers to the processing performed by that computer only.
The LOG files that may be generated by the Toonz Farm processing are the following:
server.log logs all the operations concerning the server computer activity.
controller.log logs all the operations concerning the controller computer activity.
tcomposer.log logs all the operations concerning the render activity performed by the server computers.
tcleanup.log logs all the operations concerning the cleanup activity performed by the server computers.