A Comprehensive Guide to Workload Automation

Workload automation, also known as IT workload automation, uses software to schedule, start, and run business processes, workflows, transactions, and related tasks. Workload automation focuses less on time-based processing (like classic job scheduling on siloed OS devices) and more on predefined event-driven triggers, situational dependencies, and real-time processing across heterogeneous platforms.

In addition, automation enables you to configure, initiate, or stop unattended processes; that is, tasks that are performed without constant manual intervention. Workflows, which automation configures at the command line or through a drag-and-drop functionality, describe workloads. Workload automation can work on premises, but is particularly suited to virtualized and cloud environments.

We live in a perpetually on world where data is more important than ever. Gone are the days of printing and mailing an order form or adjusting your ordering schedule to correspond with a vendor’s time zone. Automation is no longer just for the factory or IT department: When a customer needs a product, they can order it online, 24/7. Just-in-time (JIT) supply approaches thrive on data, allowing customers, retailers, manufacturers, and logistics teams to know exactly how many units of something are available or exactly how much of something is needed at each stage of the supply chain. Continuous operations boost and sustain the bottom line. Those always-available systems must also always be protected from hacks and other security threats.

As systems become more complex, fixed schedules powered by human attendants are no longer practical. Scripting across platforms is complicated and time consuming. Automation provides the answer with centralized scheduling and monitoring solutions to make business-critical systems, such as customer relationship management (CRM), enterprise resource planning (ERP), supply chain management, work order management, project management, and consulting systems work together in a timely and efficient manner.

Organizations often lack clear direction about how essential visibility into IT processes is for maintaining efficiency and cost-savings around data processing. Individual offices, departments, and groups may be allowed to purchase their own tools, selecting what seem to be the easiest solutions, such as job schedulers like Windows Scheduler. While an unrestricted approach may seem expeditious, diverse assets mean that IT staff must train on and maintain all of them. Siloed solutions also lack governance for regulatory and other compliance requirements. Although consolidation with enterprise workload automation platforms can be time consuming, the long-term cost savings and results regarding oversight and upgrades are worth it.

Enterprise workload automation capabilities enable workflows to function across diverse platforms within an organization, spanning groups and IT infrastructure. Workload automation can also be scaled down to serve smaller deployments.

As workload automation comes into its own, IT managers may have to explain to other team members — such as programmers, service desk technicians, or even the CIO, controller, or CEO — why traditional job scheduling is no longer sufficient.

In batch processing, work on large or complex transactions or system transfers is postponed and processing is conducted in large batches, often outside of regular business hours. Even as recently as 20 years ago, the limitations of computer processing and network power necessitated offline processing.

Batch processing still occurs today. For example, in a retail setting, a credit card transaction may happen in real time, but inventory control is processed in batches and happens later. Batch processing is useful when real-time analytics are not required and the high-level view of trends is more important than current returns.

A job is a set of tasks in a runnable workflow. Given the structure of a workflow, data and objects, such as document files, can be handed off between tasks, jobs, or any ultimate destination. Workflows also allow for repeatable processes, and if a process is repeatable, then it can be automated.

Job schedulers are generally OS-specific. Examples include Windows Task Scheduler, IBM iJob, UNIX, and Linux Chron. These schedulers are configured to submit batch jobs for execution according to predefined schedules. These batch jobs usually occur in the background. For instance, large print jobs may be processed through the night during non-business hours. Scheduled jobs run on one machine at a time, and, as a result, related and dependent jobs may run out of synch, with custom scripting required to ensure synchronicity.

When processes in a workflow run on different operating systems, the level of complexity increases because job schedulers rarely communicate across platforms. A human operator must manually initiate, configure, and monitor each task and process. And, this person must perform these tasks machine by machine, in what is known as attended mode. Consolidating job scheduling is possible, but requires custom scripting. In addition, compliance issues may arise if users are able to initiate sensitive and high-risk jobs in a siloed environment.

With proprietary job schedulers, you must install a client of each scheduler on any machine that talks to another system. In manual transfers of files and handoffs in workflows, errors inevitably increase. Static time and date triggers leave little room for rebalancing workloads in order to meet needs. Scaling operations to match business changes is difficult. Moreover, finding openings in a 24-hour schedule for maintenance and upgrades can be challenging, even with a pressing business case.

Workload Automation vs. Batch Processing and Job Scheduling

Workload automation offers control of enterprise business processes and multiple workflows and tasks across platforms. It supersedes batch-job scheduling by permitting the automation of unattended, event-driven workload tasks and workflows. With workload automation, processing can happen in real time and incorporate automatic resource balancing to account for special circumstances.

Managing processes and workflows through scripting is possible, but requires variables for every potentiality. Scripting in this way may be feasible for a small number of jobs, but as the number and range of jobs increase, the task becomes too complicated. When you remove the need for scripting, the range and number of processes that you can automate increase, freeing IT and other human resources for other tasks. When systems can talk easily to each other and without human intervention, errors decrease.

Classic job scheduling relies on dates and times to launch processes. With workload automation, an event usually acts as the trigger for workflows. An event can be an email, a message, an SNMP trap, the creation of a file, a webpage interaction, or something else. Removing the constraint of time-based events pushes functionality toward dynamic resource management, where systems can automatically account for priority jobs, factor in available resources, and attempt recovery from errors.

Stream processing is the younger, more advanced sibling of batch processing. Stream processing enables the analysis of data soon after it is processed. Stream processing is critical for such activities as cyber security and fraud detection, where systems detect anomalies and quickly send alerts to prevent damage from hacks or the completion of fraudulent transactions. Workload automation is essential to the effective use of stream processing.

Digital business automation describes the currently evolving model for IT automation. Whereas workload automation focuses on ensuring that web and e-commerce processes interact efficiently, digital business automation encompasses mobile and IoT workflows that rely on stream processing.

The following are the types of features that workload automation platforms offer:

• These platforms offer one view into configuring and managing end-to-end processes on a distributed system, including all platforms and all devices.

• Workload automation is often used in virtual and cloud environments. Specialized software allows centralized views and control over the many processes and transactions that occur over mainframe cloud systems.

• Workload automation platforms allow you to track active jobs, job histories, and errors, thus providing audit trails required for regulatory compliance.

• These platforms provide automated provisioning to guarantee that resources become available as needed.

• They give you the ability to see usage trends and create forecasts, while providing visualization tools to aid planning and prioritization.

• They have notifications and alerts to keep human attendants on top of the automated processes.

• They automatically transfer files between servers and processes, negating the need for manual FTP.

• They provide big data processing with no requirement for additional scripting.

The consolidation of control over platforms and processes helps standardize and simplify IT efforts and build IT maturity for an enterprise, while also reducing costs. Workload automation presents additional benefits:

• Business Benefits:

  • Automates business process workloads to improve business efficiency, returning the information you need when you need it

  • Improves your service quality for internal and external customers

  • Responds to daily business challenges, market changes, and internal company needs

  • Allows scaling across heterogeneous enterprise platforms

  • Integrates legacy business systems with new applications

  • Reduces errors caused in manual handoffs

• Customer Service Benefits:

  • Increases productivity and service quality

  • Speeds up and simplifies application delivery

  • Satisfies service level agreements with accurate forecasting and timely job completion

• Administrative Benefits:

  • Eases use and maintenance

  • Reduces labor-intensive manual changes in the system because the engine manages things

  • Centralizes control of all tasks and business processes

  • Offers a view from one dashboard into anomalies, malfunctions, and mission-critical issues across platforms and management

  • Schedules activities for optimal load balancing during high-capacity events

  • Has a user-friendly (often web-based) user interface, with mobile alerts and notifications

  • Offers a self-service capability for authorized users

  • Saves employee hours by scheduling tasks and engaging in oversight

Workload automation software simplifies and automates IT and business processes. The software centralizes your view into and management of multiple machines, platforms, and tasks, including cloud-based, mainframe, and virtual resources. It also runs batch processes, like payroll, restocking, and credit-account processing.

A workload automation framework should manage event-based, calendar, and dependency scheduling and resources across an assortment of enterprise applications, databases, and operating systems, in mainframe, cloud, and virtual environments. Here is a list of questions to ask when selecting workload automation software; the list is also available as a downloadable checklist below:

Download Workload Automation and Job Scheduling Software Checklist

Why consider a cloud resource? Your organization is probably not consistently busy — for example, a retail store might be busy on Saturday afternoons, but not so much in the middle of the night or on New Year’s Eve. Rather than buying infrastructure to meet that extra load (even though the system will sit idle most of the time), many companies invest in private, public, or hybrid cloud resources. Cloud providers usually have automation tools available that you can leverage.

Some stakeholders do have questions about cloud resources. For example, for security reasons, many still retain mission-critical, proprietary, and financial data in house, while adding less important data to the cloud. If short upload and download duration is important, data may also be retained on mainframes. Otherwise, IT managers will have to factor in extra processing time.

While by no means a comprehensive list, the following sections review a number of paid and open-source workload automation and job scheduling solutions.

Paid Workload Automation and Job Scheduling Solutions

In this section, we cover the features, product pricing, and trial offerings of various paid workload automation and job scheduling solutions.

• CA: CA, centering on CA automic workload automation, provides multiple products to address specific automation needs, all with centralized visibility into scheduling and 24/7 automation for on-premises, cloud, and hybrid environments of assorted operating systems and platforms. Features include the following:

  • Event-driven automation

  • Continuous operations during upgrades and patches

  • Multi-tenancy to permit separate business groups to share automation

  • Version control and lifecycle management to keep automation consistent throughout product development

  • Managed file transfers

  • Analytics to track processes, observe trends, and plan for high-activity periods

  • Audit logs and trails to comply with standard and regulatory requirements

  • CA Automic Sysload to provide overview analytics across platforms

  • Enterprise Job Scheduler to streamline, centralize, and optimize traditional job scheduling

  • CA Automic Dollar Workload Automation

  • CA Automic Dollar Applications Manager

  • CA Workload Automation DE (centralized control and automation with less complexity)

  • CA Workload Automation AE (centralized control and automation with customizable user interface for self-service needs and lights-out capability with automated recover and integration with Oracle EPM financial applications)

  • Support for IBM zSystems™ Mainframes with CA Workload Automation ESP Edition and CA Workload Automation CA 7® Edition

  • In a TechValidate survey conducted for CA, 68 percent of respondents preferred CA Workload Automation to BMC Control-M for viewing processes across an enterprise

  • Product Pricing: Available upon request from the vendor

  • Trial: Calculate ROI online

• BMC: BMC Control-M is marketed as a batch-job automator with centralized control for jobs across virtualized or hybrid platforms in the cloud or on premises. Control-M offers conversion for legacy schedulers and infrastructure. The solution provides analytics for IT directors and IT ops managers to monitor trends, forecast peak activity, and reduce delays. Features include the following:

  • Control-M unified batch processing for digital services

  • Secure managed file transfers for internal and external jobs

  • Control-M Self Service for control and tracking of services from mobile devices

  • Control-M for Big Data support of Hadoop

  • Control-M DevOps job-as-code for adding automated workflows to the development lifecycle

  • Control-M for Databases

  • Overview into all activities and security and resilience for operations continuity

  • Product Pricing: Available upon request from the vendor

  • Trial: Free trials available

• IBM: IBM Workload Automation provides one view into all workloads on distributed mainframe or cloud deployments. Both batch and event-driven solutions are available. The product allows you to customize dashboards for self-service portals. The tool offers segregation to meet compliance requirements and security and privacy for cloud-hosted processes. Features include the following:

  • Reduces risks to continuity and simplifies the release process

  • Easily integrates with the existing environment

  • Customizes the user interface for self-service automation

  • Includes Docker Agent for Workload Scheduler to increase capacity without adding new product instances

  • Offers customizable rerun functionality to continue jobs when they come to an abnormal stop

  • Connects jobs using variables

  • Includes REST API scheduling

  • Product Pricing: Subscriptions start at $74.30 per month for basic features

  • Trial: Fully functional 30-day trial

• Tidal: Tidal presents centralized configuration and business processes and task management across global platforms through workload automation and job scheduling in cloud, multi-cloud, and hybrid environments. More than three decades of automation experience with Fortune 1000 companies inform this workload solution, which is used around the world. Tidal ships as Oracle Cloud and is an Oracle PartnerNetwork Gold member. Features include the following:

  • Fast deployments

  • Reliability through automatic error recovery

  • A user-friendly user interface

  • Accelerated processes for cost savings

  • One view into a process

  • Product Pricing: Available upon request from the vendor

  • Trial: Free assessment of your needs

• JAMS: The MVP flagship product JAMS provides centralized initiation and management of workload automation and job scheduling across platforms, including Windows, UNIX, Linux, and OpenVMS. It is used by global organizations, including educational and government entities, software companies, and Fortune 500 companies. Features include the following:

  • Audit trails

  • A library of job templates

  • Part of a Windows package

  • Customizable dashboards

  • Product Pricing: Available upon request from the vendor

  • Trial: Downloadable 30-day free trial

• Stonebranch: Stonebranch’s Universal Automation Center provides a suite of four products: Universal Controller, Universal Agent, Universal Data Mover, and Universal Data Mover Gateway. Dynamic workflows remove the need for schedules and Universal Agents enable the management of legacy systems. Features include the following:

  • A web-based portal with mobile capability

  • For use with any scheduler on any OS (unique), any service, any platform, any technology, and any service model, including with Control-M products

  • Dynamic, real-time automation

  • 24/7 live support

  • Training through webinars and in person

  • Product Pricing: Available upon request from the vendor

  • Trial:  Free trial and online demo available upon request

Open-Source Workload Automation and Job Scheduling Solutions

In this section, we cover the features, product pricing, and trial offerings of various open-source workload automation and job scheduling solutions.

• GitHub ARM Workload Automation: GitHub ARM Workload Automation runs workloads on and gathers analytics from Android and Linux devices. This solution uses Python 2.7 or Python 3 and Linux. Features include the following:

  • Functionality implemented as plug-ins (new plug-ins easily written)

  • CustoConnect jobs using variables

  • REST Audit

  • Product Licensing: Licensed under Apache v2.0

• SOS JobScheduler: First created in 2005, this supported and mature product provides time and event-driven job scheduling. JobScheduler provides a web-based user interface for control and management of jobs that run in near-real time. The user interface, called JOC Cockpit, and REST web service are available for Windows and Linux. JobScheduler supports Oracle and SQL Server and provides agents for platforms that support Java Virtual Machines, such as Windows, Solaris, Linus, HP-UX, Mac OS, Docker, and more. This solution offers a command line or web-portal view into activities. Features include the following:

  • Runs from Linux or Windows as an unattended service, with agents for other platforms

  • Provides batch processing and near-real-time processing

  • Makes Docker Agent available

  • Makes community support available for open-source licenses

  • Product Licensing: Dual-licensing. An open-source license is available under GNU GPL (General Public Licensing). A commercial license provides enterprise-level 24/7 support, access to a ticketing portal, and the option to buy professional services.

• RunDeck: The RunDeck open source job scheduler provides compliant automation for any OS with granular security and access. RunDeck Pro offers support and additional features and plugins for successive levels of availability. Features include the following:

  • Part of Windows and Linux package

  • Secure self-service capability

  • Audit trials

  • Active development in Redwood, California, based on feedback from the user community

  • Agentless SSH-based automation

  • Product Licensing: Free, downloadable “community” version. It offers three professional levels that include live support, starting with Team for $8000 per year.

  • Trial: Free 15-day trial for enterprise version

• Activeeon: Activeeon is a flexible, scalable workload automation and job scheduling platform that meets IoT and big data needs in on-premises and cloud environments. The offering includes Automation Dashboard, ProActive Studios for building workflows, ProActive Scheduler, and ProActive Resource Manager for stand-up workflows. ActiveEon is a member of Docker, the open-source containerize platform. This solution was originally created in 2007 by INRIA, the French Institute for Computer Science. Features include the following:

  • Part of Mac and Windows to manage web, SaaS, and multi-cloud resources

  • Big data capability through Hadoop and more

  • One view into all processes and platforms

  • Audit trials required for regulatory compliance

  • Error recovery and alerts

  • Video tutorials on the product site

  • Training and support available online or in person through documentation and for those with paid licenses. The product offers three support levels for those with paid licenses.

  • Product Licensing: Available upon request from the vendor

  • Trial: Free online trial or create an account

• Schedulix: The open-source version of the commercial enterprise job scheduling software BicSuite Enterprise Job Scheduling System, Schedulix can automatically initiate, run, and monitor jobs for anywhere from a few to thousands of processes on Linux. Both the open source and commercial versions are suited to any size of operation. However, because Schedulix is based in source code, its installation requires a capable administrator. The solution is managed through the command-line interface with the Schedulix language or through the web user interface. Over 20 years old, Schedulix is fully maintained based on updates to the paid version. Features include the following:

  • Dynamic control, leveraging priorities and available resources

  • For Linux, but can be adapted to UNIX/Posix; y can run it on Windows after taking a three-day workshop with the vendor

  • Free online video tutorials

  • Community forum available

  • Available in English and German packages with downloadable manuals

  • Gold, Silver, and Copper annual support packages available for the commercial version, which offer guaranteed responses to tickets and graduated resolution times

  • Product Licensing: AGPL licence

  • Trial: Free trial

Empower your people to go above and beyond with a flexible platform designed to match the needs of your team — and adapt as those needs change. 

The Smartsheet platform makes it easy to plan, capture, manage, and report on work from anywhere, helping your team be more effective and get more done. Report on key metrics and get real-time visibility into work as it happens with roll-up reports, dashboards, and automated workflows built to keep your team connected and informed. 

When teams have clarity into the work getting done, there’s no telling how much more they can accomplish in the same amount of time. Try Smartsheet for free, today.