Main Page
From OptIPortal
OptIPortals
Introduction
What is an OptIPortal
In the broadest sense an OptIPortal is a visualization cluster which can be deployed on a variety of hardware platforms. Functionally, OptIPortals are tiled displays, that is, many displays capable of acting as one or many virtual displays. OptIPortals can be used in a wide variety of visualization approaches; viewing high definition static images, video, or in streaming mode across one or more OptIPortals. OptIPortals include both 2D and 3D environments. At Calit2, our 3D OptIPortals use OpenCOVER (by Covise) as the middleware, although we are currently seeing 3D work coming from the CGLX team as well. Our 2D OptIPortals are running either CGLX or SAGE as the middleware. OptIPortals can also be used for video conferencing and collaboration with other OptIPortals or other compatible devices (laptops, H.323 conferencing devices, etc.). The video conferencing interactions are dependent upon the OptIPortal running SAGE or CGLX, at least on as many tiles as you wish to dedicate to SAGE or CGLX. The video conferencing is achieved by using separate sending and receiving machines and then pushing the output to SAGE or CGLX running on the OptIPortal. Please see the hardware requirement section for HD Video Conferencing hardware requirements. All of these actions can be concurrent or persistent, on the OptIPortal. OptIPortal computers (nodes) comprise a cluster capable of general computations as well as display and collaboration management. Middleware Applications: SAGE, DMX, Chromium, CGLX, OpenCOVER(Covise).
Contact Information
For any questions concerning OptIPortals, please send us a note at optiportal@calit2.net.
OptIPortal Deployments
OptIPortal Deployments {images and specifications included}
System Requirements
System Software
There is considerable flexibility when building an optiportal. Depening upon your needs you can choose the OS that meets the requirements of your middleware. For example, if you will be using SAGE you will have a relatively flexible assortment of Linux flavors from which to choose. Clustering middleware is not necessary, but may make your system maintenance easier depending upon your experience. Most of the information below refers to deployments using Rocks clustering middleware in conjunction with rendering middleware, such as SAGE, CGLX and Calvr. I will be modifying this wiki over the coming weeks (completed by August 1, 2011) to include more "flat deployments" that do not require clustering middleware - feel free to contace me jkeefe@ucsd.edu for installation information concerning "flat" or cluster-mddleware deployments.
Many of our OptIPortals are built with Rocks, all software is provided via the Rocks installation - the specific OptIPortal components are obtained via the Viz Roll.
Note: after reviewing this information see the Installation and Configure Section later in this page.
Downloads
Download the Kernel Roll and Viz Roll Here
Note:
An older version of SAGE is included in the Viz Roll (updates can be found at the SAGE Downloads Site)
CGLX is available via the Hiper Roll, see the CGLX Download Site
Documentation
Rocks Installation Guide
Rocks Viz Roll Installation Guide
Middleware
This is the software that sits on top of Rocks that allows you to display images and/or video on one or more tiles.
Chromium
Available as part of the Viz Roll - see Rocks Viz Roll Installation Guide
Covise
CGLX
CGLX Home
CGLX is now available as a Rocks Roll - specifically the Hiper Roll. This can be downloaded from CGLX Site.
You can also subscribe to the CGLX Mailing List.
SAGE
Available as part of the Viz Roll - see Rocks Viz Roll Installation Guide. This version of SAGE is a bit out of date, but can be updated by grabbing the latest rpms from the SAGE Site.
Hardware
System Hardware
Since Rocks is the software environment upon which the OptIPortal is based, the hardware requirements for the OptIPortal are essentially those for Rocks. Most of the deployments of OptIPortals have been done on commodity hardware, running Intel or AMD processors. Configurations are possible in which each computer in the cluster can drive one, two or more displays depending on the performance and capabilities of the chose graphics interface. OptIPortals can be optimized for specific functionality in terms of processor speed, network bandwidth, storage capacity and memory availability.
OptIPortals whose primary application include streaming media and interactive collaboration are generally built on bandwidth balanced platforms, in which each display has general access to 50% or more of a Gigabit network interface throughput. Such systems are usually multi-core but need not consist of the fastest processors available. Each core generally has a Gigabit of memory available. The goal here is for a balanced bisection between communications speed and processor bandwidth. Computers in these systems computers generally drive 1-2 displays each.
OptIPortals that are maximized for pixel space (large display walls in excess of 30 displays or hundreds of megapixels) are usually driven by fewer computers (that is, each computer may drive 4 or more displays). As such, the computers are generally based on higher speed multi-core processors with 2 Gigabyte or more of memory each and ideally 10GigE network interfaces.
Note: if you will be connecting to the Optiputer network - be sure to use Intel Pro/1000 NICs - these gernally support Jumbo Packets (MTU 9000). The broadcom nics that come with most Dells do not support Jumbo Packets.
Network Hardware
A 1Gb switch is the minimum requirement for inter-nodal communication. We have used a variety of commodity switches for the cluster communication, including Cisco, Force10, SMC and NetGear.
OptIPuter Network
If you wish to connect to the OptIPuter network, be certain to verify network connectivity with your Network Engineer. You will need additional ports on your inter-nodal switch to accomodate the OptIPuter connectivity - depending upon which switch you have purchased, you may choose to use a second switch for this purpose. Please direct any questions to mailto:optiportal@calit2.net.
As for NICs, use Intel Pro/1000 whenever possible - this will make certain that you can use Jumbo Packets (MTU 9000) if you will be connecting t o the Optiputer network.
Graphical Hardware
Displays
Most displays suitable for computing environments will operate, however, if you will be displaying high definition imaging, it is best to use a relatively modern LCD display for your displays. We have typically used 24", 30", 46" and 52" LCDs.
Graphics Cards
Be sure to choose a graphics card that can drive your displays at the desired resolution. You will have to balance heat and power requirements for the systems you have selected (and for the location where the OptIPortal will reside). Proper heat balancing is essential to the longevity of the systems driving the displays. Recently, we have been using nVidia GTX 480 and 580 card. Choose the model that best meets your applicatoin needs (and that doesn't exceed your hardware specifications).
Framing
The vendor we have used mosted often for OptIPortal framing is a company named 80/20.
Over the past few months, we have been using a portable 2x2 display configuration. We will have an order spreadsheet available in the near future. Greg Dawe, UCSD Calit2, has been the chief engineer on this project, he can be contacted as follow:
- Greg Dawe
- gdawe@ucsd.edu
Bezel widths are getting smaller every year, we have been using NEC 46"' Narrow Bezel displays for our most recent deployments, including the 2x2 portable setup above. Note: be certain to carefully read the ordering instructions on the spreadsheet - you will be ordering by number of stems (columns) required for your configuration.
How to Assemble the Frame
NEC 46-inch Narrow Bezel Parts List Spreadsheet
Framing Supply Contacts
- Mike Fanolla
- F & L Industrial Solutions, Inc.
- (858) 271-6732
- fandlindsol@sbcglobal.net
HD Video Conferencing
HD Video Conferencing is achieved by setting up separate sending and receiving machines and then pushing the output to the OptIPortals using one of the two following methods (for both methods our current recommendation is to use a commercial VTC option, such as LifeSize - see the "Commercial VTC Option" below).
- CGLX Capture System: to use this option a PC with a BlackMagic card is placed on the private internodal network (physically connected to the internodal switch) - the video out signal from the QVideum box is connected to the BlackMagic card - the stream is then sent to the OptIPortal and can be displayed via the StreamWindow in the MediaViewer app in CGLX.
- SAGE Capture System: this system is quite similar to machine above, PC with BlackMagic card, the SAGE machine runs Opensuse instead of Windows: to use this option the OptIPortal must be running SAGE to receive the output from the HD Video Conferencing machine. The output is then pushed to the OptIPortal using a SAGE Capture System.
HD Video Conferencing Hardware
Commercial VTC Option
A commercial HD VTC solution, such as LifeSize can be used. Here at Calit2, UCSD, we have used the LifeSize Room 200 series - LifeSize example. We then connect the video output to the streaming box (described in the prior section) and direct the output to the wall. The LifeSize Room 200 is just one example of other similar commercial products. The advantages are simplicity and compatibility with other VTC hardware and complete audio support.
Research Option
The research option research grade video delivery, the bandwidth requirements tend to be greater and will require more system administration support. Here is a recently deployed example, Ultragrid. Todd Margolis, UCSD Calit2, has been doing work with Ultragrid, he can be contacted as follows:
- Todd Margolis
- tmargo@gmail.com
Installation and Configuration
How do I build a tiled wall or OptIPortal
There is tremendous flexibility in the hardware to be used. See Recent Deployments for a recent list of installations.
The OptIPortal is built with Rocks, all software is provided via the Rocks installation - the specific OptIPortal components are obtained via the Viz Roll.
Example Deployment
Bill of Materials (BOM)
http://wiki.optiputer.net/optiportal/images/6/68/OP-classic-bom090811.pdf
Download the Software
Download the Kernel Roll and Viz Roll Here
SAGE is included in the Viz Roll - be sure to add the Hiper Roll from http://vis.ucsd.edu/~cglx - you will then be able to choose the middleware that best suits your needs (review the middleware choices earlier on this page).
Download the Hiper Roll Here
Note: download the kernel and viz roll iso's, boot from this kernel roll CD on your head-node, type front-end at the prompt, you will soon see a screen asking you to Select Your Rolls - select Download from the Network-based Rolls. Be sure to select the following rolls: kernel, base, hpc, os, viz web-server. If you have downloaded the Hiper Roll, be sure to select CD/DVD from the Local Rolls section of the Select Your Rolls screen before proceeding.
If you cannot download network based installation rolls, then when you burn media be sure to add the individual OS roll media in addition to the jumbo DVD. Installing from burned media is fine, but the OS roll information, specifically, will be more complete when doing the network based roll.
Be sure to review the Installation Documentation.
Rocks Installation Guide
Rocks Viz Roll Installation Guide
Installation FAQ
SAGE Configs for 2x2 Optiportable
These configs are for a 2x2 (4 NEC 46" Narrow Bezel) optiportal.
- Setup ssh keys
- run sage to setup template directory .sageConfig in demo home, you should get a green SAGE Startup Screen - press Start and the black UI will appear. If you do not see the black SAGE window, check the terminal logs for failed ssh logins - exit sage and reverify your keys.
- Having run sage for the first time, you will see a .sageConfig directory in your home directory. Place the following two files in this location:
{ROCKS only} Using a USB attached DVD drive to install the frontend
Append driverload=usb-storage to the build command when you first boot the frontend. {taken from the Rocks Wiki}
{ROCKS only} Disable auto-reinstall of tile nodes
tentakel /etc/rc.d/init.d/rocks-grub stop
tentakel /sbin/chkconfig --del rocks-grub
{ROCKS only} Force Reinstall at Next Reboot
Occasionally, you may wish to reinstall a node at its next reboot; for example, replacing a bad disk.
Example:
rocks set host boot tile-0-3 action=install
Then reboot tile-0-3
Be sure to disable auto-reinstall (if you prefer) after rebuild. See: Disable auto-reinstall of tile nodes
Add broader multicast route for CGLX
For CGLX to function properly create the following to each node, including the head-node - assuming eth0 is the interface for your 10-net:
/etc/sysconfig/network-scripts/route-eth0
The contents should be:
NETMASK0=240.0.0.0
ADDRESS0=224.0.0.0
Restart the network on each node and head-node.
{ROCKS only} root ssh keys not passed to node at installation
Periodically you may find that you are prompted for a password as root when going from head-node to tile-node.This has typically happened after an upgrade installation.
- Short-term fix:
- copy /root/.ssh/id_rsa and /root.ssh/id_rsa.pub
- be sure to follow the procedure below for installation of new nodes
- Long-term fix:
- you need to make '/root/.ssh/id_rsa.pub' world readable, then
- reinstall your compute nodes.
- # chmod a+r /root/.ssh/id_rsa.pub
{ROCKS only} Head-node - failed X server
Occasionally with the Viz roll, the head-node will often have an X Server failure at first startup after installation - simply remove /etc/X11/xorg.conf and then regenerate - use nvidia-xconfig or example and then restart X (init 3; init 5).
Update nVidia Driver
Download the latest nVidia Driver from http://www.nvidia.com. Place in /share/apps and make the installation script executable. For example:
head-node:
init 3
/share/apps/NVIDIA-Linux-x86_64-185.18.14-pkg2.run --no-network -s > /dev/null
init 5
tile-nodes:
tentakel init 3
tentakel /share/apps/NVIDIA-Linux-x86_64-185.18.14-pkg2.run --no-network -s > /dev/null
tentakel init 5
xorg.conf for CGLX
- Make certain that all nodes have the following in their /etc/X11/xorg.conf (including the head-node):
- Section "Module"
- Load "vnc"
- Load "dbe"
- Load "extmod"
- Load "type1"
- Load "freetype"
- Load "glx"
- EndSection
{ROCKS only} Tile Layout: Rocks Method - from Viz Roll Docs
The Viz Roll supports the construction of walls with either a single LCD display per machine or multiple displays per machine. Multiple displays can be grouped together as a single X11 display using TwinView with or without Xinerama, or each display can be an independent X11 display. The choice of whether to group a host's physical displays together or not is controlled by the rocks sync tile command.
The default configuration for Rocks is a single physical display per machine. If this is your configuration skip to the next section. If your wall has multiple displays per host you will need to create an XML an XML file that describes the physical layout of your wall. The syntax is very simple and the XML itself maps to the dimensions of the wall. Each col tag describes a vertical column of the wall, with the nodes listed top to bottom. The very first col section lists the Tile nodes comprising the far left side of the wall, and from there the next section moves to the right.
Warning
When connecting a single video card to two display you must connect the primary video connector to the LCD above (or to the left) of the secondary connector. This is a requirement of the SAGE software.
1. This simplest method to create you layout XML is to start from scratch rather than modifying the defaults as is documented in the non-twinview section above.
<wall>
<col>
<display host="tile-0-0:0.0"/>
<display host="tile-0-1:0.0"/>
<display host="tile-0-2:0.0"/>
</col>
<col>
<display host="tile-0-0:0.1"/>
<display host="tile-0-1:0.1"/>
<display host="tile-0-2:0.1"/>
</col>
<col>
<display host="tile-1-0:0.0"/>
<display host="tile-1-1:0.0"/>
<display host="tile-1-2:0.0"/>
</col>
<col>
<display host="tile-1-0:0.1"/>
<display host="tile-1-1:0.1"/>
<display host="tile-1-2:0.1"/>
</col>
<col>
<display host="tile-2-0:0.0"/>
<display host="tile-2-1:0.0"/>
<display host="tile-2-2:0.0"/>
</col>
</wall>
2. After you update your tilelayout XML, now apply it to the database:
# rocks add tile layout layout.xml
3. Finally, reconfigure your tiles:
# rocks sync tile mode=meta
Tile Layout: Creating Customized /etc/X11/xorg.conf
In some cases you may want to customize the xorg.conf on the nodes - many opencover and cglx users prefer this method - but you will have to keep track of your modifications. The easiest method is to run nvidia-settings from the nodes and display the gui on the head-node:
open a root shell on the head-node - ssh to tile-node and run nvidia-settings on the tile-nodes, for example:
head-node# ssh tile-0-0
root@tile-0-0# nvidia-settings --ctrl-display=tile-0-0:0
{save your changes and restart X}
SAGE Startup Issue
- First time startup of SAGE gives: "UnboundLocalError: local variable 'tile' referenced before assignment"
- you must first run the tile setup first: see steps above:
- File_Server startup problems:
- stop and start via advanced window in SAGE startup ui - click on the red X to toggle to green check
- try starting from command line /opt/rocks/bin/python2.4/file_server.py
How to change Public IP on head-node
This is not a complete process, but will work reasonably well for general cluster behavior.
Use the following:
* rocks set host interface ip cluster.domain eth1 <new-IP>
* rocks sync config
If you see python errors from rocks sync config - manually change /etc/sysconfig/ifcfg-eth1 and restart networking - /etc/init.d/network restart - wait a couple of minutes and rerun rocks sync config.
Note: this process is incomplete, but may suffice for most cluster operation, you will still see the old IP on node Kickstarts - to change this try:
* rocks set var Kickstart PublicAddress <new-IP>, this invocation looks correct, but was failing under Rocks 5.2.
The absolute robust way is to prepare a restore roll and re-build using the new IP.
What if you don't have a Public IP for the head-node
We call this road-show mode. Of course, you will need all of your installation media - Rocks Jumbo DVD, etc.
You can just assign a dummy IP address for the Public IP at installation time. After you have PXE booted all your nodes, ssh to each node (there will a delay {dns induced}) and change the IP of the head-node Public to the head-node's 10-net address. The nodes will now communicate properly with the head-node and all will be well. A kluge to be certain, but functional.
SAGE libGLEW Version Mismatch
- When using SAGE after having installed CGLX - there will be libGLEW mismatch - SAGE is looking for libGLEW 1.4, CGLX uses 1.5. Correct by creating 1.4 link to 1.5 lib:
- [head-node] ln -s /usr/local/lib64/libGLEW.so /usr/local/lib64/libGLEW.so.1.4
- [tile-nodes] tentakel ln -s /usr/local/lib64/libGLEW.so /usr/local/lib64/libGLEW.so.1.4
SAGE VNCViewer
The SAGE VNCViewer is a great tool for flexible on-screen collaboration with any laptop (or desktop) that can support VNCServer. When installing VNCServer under Windows 7, be sure to deselect installing as a service during the installation. There are a couple of other superstitious modifications to the VNCServer config - use the legacy settings; 3.3 settings, etc.
