Job Scheduling Maze in Distributed IT Landscapes - Part 3




Part 2 of this blog series analyzed the ActiveBatch architecture and evolution in terms of enterprise job scheduling and workload automation functional capabilities. This was done for the three older product releases. Particularly impressive was the ActiveBatch V6 release that introduced a few noble concepts like Job Library and Virtual Root.

ActiveBatch V7 – The Current State of Affairs

The ActiveBatch V7 release (the release has completed the Beta phase, and is now in the final quality assurance [QA] phase; its release date is scheduled for January 2009) is not a slouch either when it comes to introducing new concepts like the one of Dynamic Web Services within the Business Process Automation (BPA) module. This module allows both externally and internally developed Web Services to be incorporated into workflows, and to trigger workflow steps.

These specific business objects can be created on the fly to combine ActiveBatch services (e.g., a trigger service) with any Web Services-enabled application to allow for jobs and job plans to retrieve information from external or internal data sources and applications. This feature allows the resulting job step to be re-useable in multiple workflows, and to facilitate the creation of composite applications.

The integrated and extensible Job Library capability is also pertinent here via stored routines for applications and infrastructure to developers and users and to the overall business. The library supports  Microsoft .NET Framework class library and Web Services architecture, and the V7 release enables inbound and outbound integration of jobs, notifications, and triggers. New events were introduced for email triggers, such as using key words, Microsoft Message Queuing, Web Services, and more.

Helping with ITSM and “Green” Needs

In addition to its Web Services and service oriented architecture (SOA)-based integration and process modification capabilities, two other notable improvements debuted in V7. First, a Change Management/Information Technology Service Management (ITSM) system with a new user interface (UI) was introduced to allow the reliable and fast movement of ActiveBatch objects between the Development, Quality Assurance (QA), and Production departments. To make it easier to update and manage ActiveBatch objects, V7 supports the creation, comparison, modification, and updating of objects from one environment to another.

For example, objects can be moved from a QA environment to a production environment in a simple and auditable fashion. The environment enables synchronization across job schedulers in different departments and in different lifecycle phases, whereby object differences can be highlighted and color-coded. Users can select whether to apply changes later (save for approval), apply changes now (based on permissions), or to enable a controlled time window for changes.

All these capabilities, together with objects’ dependency resolutions, aim to provide greater reliability when managing multiple and diverse computing operations. Modifications to objects can be compared across revision levels to see what changes were made and by whom. The designer can elect to rollback to a previous revision should this be required, while custom callbacks for object customization can be applied to all objects, job user accounts, queues, etc.

Second, given the increasing calls for energy preservation everywhere, the V7 release introduced a “green” Power Management feature that allows idle machines to be placed in a Hibernate or Suspend state to save on power consumption and potentially take advantage of rebates or passing carbon credits. The dormant systems can be reactivated either at a specific time or via the hardware’s “Wake-On-LAN (WOL)”  function.

Moreover, the separate graphical view enables monitoring of service level agreements (SLAs), in terms of service level violations, potentials for violation, successes/failures, and whether the business process succeeded even with a SLA failure. There is also integration with business rules engines (e.g., for completion rules creation and dynamic runtime prioritizations).

Rule constraints can be set in terms of jobs, files, resources, and other variables. Other features include the ability for users to query jobs’ execution state and to evaluate idle times. However, the SLA capabilities have been pushed out to what will likely be a Service Pack (SP) release after the January 2009 release date of ActiveBatch V7.

From the performance standpoint, release V7 features improved scheduling through dynamics characteristics of a historical data-driven Online Analytic processing (OLAP) database. The database is constantly populated by historical data details to develop “hints” for optimized performance (e.g., to minimize elapsed time, optimize resource utilization, etc.) and to forecast server loads.

New and/or Enhanced Job Types and Views

As an additional job type alternative (to the five types explained in Part 2), the Job Library was introduced with V6 and is being expanded in V7. The Job Library contains templates of job steps for applications such as Crystal Reports, Symantec Veritas Backup Exec, multiple database functions, ActiveBatch Task Scheduler jobs, Power Management, and so on.

The Library Job Steps can be dragged and dropped to create a workflow. The user has simply to click on the provided action template, drag it to the Job Step Editor, and complete the fields, some of which use discovery to provide appropriate choices (e.g. in those systems where Microsoft SQL Server database is installed, and it enumerates the Structural Query Language [SQL] jobs, etc.)

Also, in ActiveBatch V7 the existing two views, System View and Design View, were combined into a single graphical view. Workflow designers can use this visual approach to create and link jobs as well as plans together into "end to end” workflows using completion triggers, alert triggers.

They can also manage them by implementing Constraints or Dependencies between them using drag-and-draw operations. This view also allows operations to graphically review not only the workflows as they are triggered, but also the progress from job to job as well as any status change, to visually alert operations should there be an issue requiring actions such as a re-start, re-run, force completion, etc.

The Graphical System View lets users see the current status and immediate results of every job in the queue. Icons in the view show job dependencies and workflow, and jobs are color-coded based on the current run status and the last execution result (e.g., a purple color denotes a failed job).

Why ActiveBatch Wins Customers

ActiveBatch’s parent, Advanced Systems Concepts Inc. (ASCI), touts the differentiating features like ActiveBatch Mobile Management (with full remote management of the system from SmartPhone and Personal Digital Assistant [PDA] devices), incorporating SOA for increased levels of integration, change management, etc. In addition to being inexpensive (an average price of deployment is about US$ 35K), ActiveBatch is also relatively easy to deploy and use because its architecture includes many key components that are typically already deployed in the customers’ IT infrastructure.

Included in this set of supported technologies are the following: Windows Security Model, Active Directory and Active Directory Application Mode (ADAM), Microsoft SQL Server and Oracle databases, Windows Management Instrumentation (WMI), Simple Network Management Protocol (SNMP) trapping, and so on. This familiarity-based ease-of-use, script language independence, and the ability to integrate to existing IT infrastructures (either via Microsoft Common Object Model [COM] or Web services) reduce the learning curve and the overall cost of operations for customers.

ASCI admits that the customers' decision to implement ActiveBatch over a competitor rarely comes down to a features/functions-based selection, but rather on the overall issues like the ease of installation and use of the product, price, scalability, and reliability.  The vendor often finds that, when users look at products like ActiveBatch, they have immediate needs, longer-term needs, and some conceptual needs (i.e., “We like that, and may find use for it in the future, so it is nice to know that it is there”).

Compared to many of the competitors mentioned in Part 1, especially those that originate from mainframe environments, ASCI’s decision to directly incorporate technologies in ActiveBatch (rather than to simply tolerate or retrofit them) gives it a strategic advantage. For example, the technologies like Windows Security Model with the Kerberos authentication protocol, ADAM (and even enhanced directory services for non-Windows platforms), cluster servers (both Microsoft’s and Symantec Veritas Cluster Server), Oracle and SQL Server databases, and so on are all integral to ActiveBatch, but rarely to other vendors’ offerings.

When it comes to scalability, as already emphasized in Part 1 and Part 2, ActiveBatch has tested up to 2,000 server connections and over 1.2 million jobs in 24 hours.  Such architectural limits are showstoppers for many if not all of competitive products.  In addition, the Virtual Root capability for multiple occupancy of ActiveBatch Job Scheduler (which was introduced in V6 and explained in Part 2) allows for full and secure (via directory services) transparency between users, departments, divisions, etc.

In addition, ASCI has adopted a sales model of disintermediation, thus centralizing its operations with aggressive use of the Internet for working with clients around the world from its United States (US) headquarters (HQ). This includes utilizing both the ActiveBatch sales and support experts, and customers’ direct access to the engineering group.

The net result is a more functional product and better customer service through improved information flow with dramatically lower costs of sales and support, where the savings can be passed onto clients. Today’s Internet allows precisely for this new approach to selling and supporting enterprise applications.

Going SaaS/On-demand or Not?

This brings us to the fact that the current ActiveBatch architecture lends itself well to the software as a service (SaaS) model. In fact, ActiveBatch Job Scheduler supports multi-tenant private/secure occupancy through the Virtual Root capability. ASCI believes that ActiveBatch’s Web-based (thin) client (in addition to mobile access to the system, and powerful backend database offerings of Oracle and SQL Server) can handle the required performance. The rapid and lightweight deployment strategy for the ActiveBatch Execution Agents makes it also easy to address firewall and other requirements.

So, while ASCI does not offer a SaaS option for ActiveBatch at this time, it understands the issues, and is researching how and when this approach to the market can offer the greatest rewards for the vendor and its clients. ActiveBatch’s focus for its upcoming release is also focused on SOA-based integration as opposed to business process management (BPM).

The business process orchestration that includes both manual processes and human intervention, as used in the BPM suite sense is not the market ASCI is going after, at least not now.  The ActiveBatch V7 release will certainly integrate with other SOA governance/BPM suites, and is compatible with many SOA/BPM applications.

Parting Conclusions

Despite its functional and architectural superiority (in terms of system availability and performance benchmarks, and versatile administration features), and the price advantage (being usage-base licensed by the number of jobs per month, while Web hosting reduces the cost of operations for the vendor), the customers’ word of mouth still largely sells ActiveBatch. The company needs to invest in much more aggressive marketing and prospective customers’ education, despite the large prospective audience and a wide open playing field.

Based on data collected over the last several months by ASCI, which could provide a reasonable indicator of ActiveBatch uses, two major deployment areas have been for Data Warehousing  and Business Intelligence (BI), and then Enterprise Application Integration (EAI)  of new, legacy, and/or packaged applications.

One possible fertile ground for ActiveBatch to pursue would be to educate prospective users to better understand and consider the addition of powerful scheduling and automation capabilities to their existing enterprise applications sets like supply chain management (SCM), customer relationship management (CRM), BI, etc. These commercially available applications might require products like ActiveBatch to allow them to be integrated into and across workflows as well as to add scheduling techniques to the more rudimentary scheduling found within these applications themselves.

A good case study would be a semiconductor manufacturer with 15 disparate enterprise resource planning (ERP) systems in its plants running on Microsoft Windows and Sun Solaris platforms. An overarching SCM product from i2 Technologies is used to create an overall multi-site master production schedule (MPS).

Generating a corporate-wide MPS involves processing more than 100 jobs, all of which must be performed in a precise sequence. Given so many dependencies, the failure of one job would affect the true optimization of the final MPS plan; it is thus better to repeat the failed job (after investigating and fixing the cause) then to get the suboptimal overall plan. ActiveBatch came in handy in that regard, since the manufacturer attributes to it to a 50 percent reduction of the planning staff, 30 percent in inventory reduction, and 10 percent in cycle time reduction.

To that end, ActiveBatch might benefit from having ready-made connectors to leading ERP products. Its competitor Redwood Software has long had a custom solution using a “connector” aimed specifically for SAP Central Process Scheduling and SAP Solution Manager. ActiveBatch might provide a more powerful framework, but not the SAP connector at this time. The vendor would currently use its SOA Dynamic Web services or Command Line Interface (CLI) for a general interface to a SAP.

ActiveBatch’s plan is to develop an SAP connector for ActiveBatch to be released in 2009. Meanwhile, the ActiveBatch  service library allows the vendor to connect applications using web services, expose the methods, and make use of their properties. The created service is then associated with a job as part of the Job Library. The goal is to allow any application or system that can use web services to integrate and trigger any other application.

Therefore, dear readers, what are your views, comments, opinions, etc. about ActiveBatch’s applicability, and abut the enterprise job scheduling and workload automation software markets in general? We would also be interested in your experiences with this software category (if you are an existing user), and your general interest to evaluate these solutions as prospective customers.
 
comments powered by Disqus