Basics of Oracle AQ

Oracle Advanced Queue, as the name suggests has all the queue features. The added advantage which oracle AQ offers is the oracle DB, means we can have a persistent queue which will be stored in db.

Oracle AQ follows some basic structure to store the message in queue.
first of all we need to create AQ table which will be having queues and then can schedule propagation and create subscribers.

Just to quote the definition from Oracle
 "Oracle Advanced Queuing (AQ) is a database-integrated messaging infrastructure
in Oracle Database 11g. AQ leverages the functionality of the Oracle database to
store messages in persistent queues. All operational benefits of the Oracle database
such as High Availability, Scalability and Reliability are applicable to the messages
and queues in AQ. "



When I had to work on AQ the basic confusion I had was
how propagation and subscribers are different ?
and when to use what ?

well the basic difference that I understood was, Creating subscribers is a way to tell the corresponding queue can be dequeued by the declared subscriber. So that if there is any rule or transformation required for that subscriber then it will be done while dequeuing operation only.

Propagation is pretty straightforward. As soon as message comes or as per settings(time and delay) the message will be propagated to other queue, this queue can be in the same database or it exists in some remote db.

Sample troubleshooting steps:

1.      1.  Check if the database link to the destination database has been set up properly. Make sure that the queue owner can use the database link. You can do this with:

Select count(*) from table_name@dblink_name;

2.       2. Check whether Queues are enabled for enqueue/dequeue:

Select * from USER_QUEUES where name =  'QUEUE_NAME';

2.1            If the queues are disabled for enqueue/dequeue, then we can start it by using following sample command:

EXECUTE DBMS_AQADM.START_QUEUE(QUEUE_NAME => QUEUE_NAME');

commit;

3.       3.  Check messages in the queue:

select *  from AQ$QUEUE_TABLE_NAME;

4.      4.  Check the subscribers for  the queue :

select * from ALL_QUEUE_SCHEDULES;

Common AQ Errors(copied from Oracle docs):

ORA-1555

You might get this error when using the NEXT_MESSAGE navigation option for dequeue. NEXT_MESSAGE uses the snapshot created during the first dequeue call. After that, undo information may not be retained.

The workaround is to use the FIRST_MESSAGE option to dequeue the message. This reexecutes the cursor and gets a new snapshot. FIRST_MESSAGE does not perform as well as NEXT_MESSAGE, so Oracle recommends that you dequeue messages in batches: FIRST_MESSAGE for one, NEXT_MESSAGE for the next 1000 messages, then FIRST_MESSAGEagain, and so on.

ORA-24033

This error is raised if a message is enqueued to a multiconsumer queue with no recipient and the queue has no subscribers (or rule-based subscribers that match this message). This is a warning that the message will be discarded because there are no recipients or subscribers to whom it can be delivered.

ORA-25237

When using the Oracle Streams Advanced Queuing navigation option, you must reset the dequeue position by using the FIRST_MESSAGE option if you want to continue dequeuing between services (such as xa_start and xa_end boundaries). This is because XA cancels the cursor fetch state after an xa_end. If you do not reset, then you get an error message stating that the navigation is used out of sequence.

ORA-25307

Flow control has been enabled for the message sender. This means that the fastest subscriber of the sender's message is not able to keep pace with the rate at which messages are enqueued. The buffered messaging application must handle this error and attempt again to enqueue messages after waiting for some time.

This white paper contains all the basic info about AQs:

http://www.oracle.com/technetwork/database/features/data-integration/oracle-aq-tech-wp11-2-191324.pdf

Fundamentals of BPEL

To Start with, BPEL (Business Process Execution Language), is an OASIS standard executable language for specifying actions within business processes with web services. Source Wiki

So if we say BPEL we mean a standard language, whose basics will not change as per the implementing companies/product. All we need is BPEL engine to run the BPEL code. This engine can be open source or licensed one, depending on the additional features we may require for the product/service to be developed.

For example we have few companies IBM, Oracle, WSO2 which strictly follow BPEL (OASIS) standards and provide additional features on top of the basic BPEL.

Latest version of BPEL is 1.2 -->2.0 which was revised from 1.1.

Few differences are :

New Activities 2.0

• activity to repeat the set of activities . Replace the FlowN activity in BPEL 1.1 version.

• - Use this activity if the body of an activity must be performed at least once. The XPath expression condition in the repeatUntil activity is evaluated after the body of the activity completes.

•  -- - Replaces the switch activity in BPEL 2.0 - This activity enables you to define conditional behavior for specific activities to decide between two or more branches. Only one activity is selected for execution from a set of branches.

• Changed to

•  - to compensate the specified child scope

• A activity has been added to fault handlers this activity enables you to rethrow a fault originally captured by the immediately enclosing fault handler.

New activity types:- repeatUntil, validate, forEach (parallel and sequential), rethrow, extensionActivity, compensateScope 
Renamed activities:- switch/case renamed to if/else, terminate renamed to exit 
Termination Handler:- added to scope activities to provide explicit behavior for termination
Variable initializationportType attribute:- The attribute "portType" of messaging activities such as "receive," "invoke," "reply," "pick," and "onEvent" is no longer mandatory and can be omitted

Source: http://www.albinsblog.com/2012/09/difference-between-bpel-11-and-bpel-20.html#.VR5s3cbCkRY

Just To summarise and revise ;) 

Messaging:- In BPEL 1.1 there is no defined behavior for a process that receives a message for a request-response operation and finishes without replying. In BPEL 2.0, such a scenario would have trigged a new BPEL standard fault called "missingReply" fault.

Source : http://pushplsingh.blogspot.in/2011/10/comparison-between-bpel-11-and-20.html

Coming back to BPEL, since I am from Oracle SOA suite background I would share my learnings in BPEL in terms of features and extensions provided by oracle.

Oracle extensions: 

Email 

Java Embedding activity:

we can embed Java code snippets directly into the BPEL process using the Java BPEL exec extension.

When we drag a Java Embedding activity into a BPEL process in Oracle BPEL Designer, the  element and bpelx:exec tag are automatically added.

IM

Replay

Transform

Dehydration Store :As discussing this will require more than one blog..there is one informative blog for dehydration you can refer to : http://www.soatutor.com/2012/08/dehydration-in-bpel-oracle-soa-suite-11g.html

Dehydrate

Over the life cycle of a BPEL instance, the instance with its current state of execution may be saved in a database. When a BPEL instance is saved to a database, the instance is known as being dehydrated. The database where the BPEL instance is saved is called a dehydration store.

Skipping dehydration

• For Synchronous processes that do not need to be durable, you can turn off the dehydration mechanism by setting the following properties at process or engine level:

• Skipping dehydration will increase performance with a cost of auditing.

• By Default, all BPEL processes are dehydrated regardless of whether they are Synchronous or Asynchronous process.

• Set inMemoryOptimization to true.

• Set completionPersistPolicy property to faulted or Off.

• Asynchronous process invocation messages are always saved to the dehydration store.

• Durable process are always dehydrated regardless of the settings of the persistence properties.

Dehydration Tables

1. Cube_Instance: Stores the information about the composite instance that gets created.
2. Cube_scope: Stores information about the scopes and variables declared and used.
3. DLV_Message: All the incoming payload details are stored.
4. Invoke_Message: All the outgoing message payload details are stored.
5. Audit_Trail: Used to store information of the xml that is rendered in EM console.

Link Referred :http://manoharsr.wordpress.com/2013/05/13/dehydration-in-soa/

OSB Throttling and Service Virtualization and other concepts

Message Throttling, Service Virtualisation ,Load Balancing and Service Result caching features of OSB are some of the main features of OSB which gives it an added advantage over other integration tools..


Message Throttling -

A common problem with integration is the risk of overloading a particular web service. When the capacity of a web service is reached and it continues to accept connections, it will most likely start to deteriorate.

Limiting the Concurrent Number of Requests

Limiting the concurrent requests for a Business Service cannot be set at design time so you have to use the built-in Oracle Service Bus Administration Console to do it (/sbconsole). Follow these steps to enable it:

In Change Center, click Create to start a new Session

Select Project Explorer, and navigate to the Business Service you want to limit

Select the Operational Settings tab of the View a Business Service page

lIn this tab, under Throttling, select the Enable check box. By enabling throttling you

pSpecify a value for Maximum Concurrency

pSpecify a positive integer value for Throttling Queue to backlog messages that has exceeded the message concurrency limit

pSpecify the maximum time in milliseconds for Message Expiration a message can spend in Throttling Queue

lClick Update

lClick Active in Change Center to active the new settings

Service Virtualisation - 
Service virtualization is an approach to deploying and managing services.

A core principle of SOA is to ensure that any service consumer can access any service provider - and from any platform. This has been considered as key principle in OSB and it provides robust way of Vitalizing the Service. It’s a great value add in SOA Architecture.

This we can achieve in OSB, by doing,  protocol transformation, message transformation and by routing the service to the end system(backend) dynamically. 

Service Result Cache - 

Service Result Caching is one of the options that you can use when you want to improve Oracle Service Bus performance. Service Result caching is used when we have business service which connects to external service which returns somewhat static response. So by using Service Result Caching we don’t hit external service for same request instead it will take the response from cache which improve the OSB performance

Load Balancing:

lIn Oracle Service Bus you must define at least one endpoint URI for a business service. When you define multiple endpoint URIs for a business service you must define one of the following load balancing algorithm:

pRound robin

pRandom

pRandom-weighted

pNone

lThe load balancing algorithm controls the manner in which business service tries to access the endpoint URI. 

Round-robin - This algorithm dynamically orders the URLs that you enter in the Endpoint URI field for this business service. If the first one fails, it tries the next one, and so on until the retry count is exhausted. For every new message, there is a new order of URLs.

Random - This algorithm randomly orders the list of URLs that you enter in the Endpoint URI field for this business service. If the first one fails, it tries the next one, and so on until the retry count is exhausted.

Random-weighted - This algorithm randomly orders the list of URLs that you enter in the Endpoint URI field for this business service, but some are retried more than others based on the value you enter in the Weight field.

  None - This algorithm orders the list of URLs that you enter in the Endpoint URI field for this business service from top to bottom.

Some quick FAQs :

1. when you deploy the service does caching get refreshed ?
Ans: It will not be refreshed, when proxy service is redeployed. But will be refreshed when BusinessService(where actual caching is happening) is deployed.

2. When you change the service through console and then redeploy the service from eclipse does it overwrite the changes like monitoring and throtlling, endpoint uri  ?
Ans : It will overwrite endpoint uri.
Monitoring - no
Throttling - no

As Monitoring and throttling are done from SB console these values will remain as is.

3. For Load balancing when we set offline URI option for the unavailable business service , Will offline URL will be offline after deployment ?
Ans. yes it will be offline, till its time is over or we make it online through console.

4. Load balancing algorithm in OSB Business service ?
 Round-robin - This algorithm dynamically orders the URLs that you enter in the Endpoint URI field for this business service. If the first one fails, it tries the next one, and so on until the retry count is exhausted. For every new message, there is a new order of URLs.

Random - This algorithm randomly orders the list of URLs that you enter in the Endpoint URI field for this business service. If the first one fails, it tries the next one, and so on until the retry count is exhausted.
Random-weighted - This algorithm randomly orders the list of URLs that you enter in the Endpoint URI field for this business service, but some are retried more than others based on the value you enter in the Weight field.
None - This algorithm orders the list of URLs that you enter in the Endpoint URI field for this business service from top to bottom.

5.  In result caching, what is the TTL(Time to live) by default?
Ans; Default expiration time is taken from OSB coherence config file- //config/osb/coherence/osb-coherence-cache-config.xml
By default it is set to 5 minutes,

6. In Result caching how and where we use xquery expression for  Expiration time ?
Ans. It is worth noting that the Cache Token may be specified as any expression you like, using the XQuery functional programming language.
XQuery has a rich syntax for constructing queries, including the use of many system libraries with common functions such as numeric operations and string manipulation.
Clear the cache explicitly
It might be that you can’t risk the cache becoming stale under certain circumstances.  If that is the case, rather than using a time parameter to manage the lifetime of the cached responses, mark critical requests with an additional flag and select XQuery Expression in the expiration time options to test for the existence of this flag.

7.where we will set the persistent level and priority while calling JMS queue ?
Ans: In JMS transport page of Business service we can set the message persistence.

For Message Priority, we can use transport header activity in Proxy service to set the priority for a message



8. what is this Tracing option in monitoring page ?

By using tracing option, we can actuallly see the message flow for that particular service in logs.Tracing messages in the Oracle Service Bus can give you a very good insight of what’s really happening inside your services. Message tracing can be very valuable for detailed monitoring the content of the requests (full) and give you insight on the amount of service calls going in your service (headers or terse).

The tracing information is stored in the server directory logs at the following location:

DOMAIN_HOME/domain/servers/server_name/logs/server_name.log

Source: http://jvzoggel.wordpress.com/2012/01/23/osb_tracing_runtime/

9. In Sbconsole, inside General Configuration, what is the use of state ?
Ans. It actually say whether the service is online(available) or offline.



Very useful brief Usage of operational settings,
"Copied From Oracle " - http://docs.oracle.com/cd/E28280_01/admin.1111/e15867/monitoring_ops.htm

Operational Settings for Services in the Administration Console

Operational Settings	Usage	Default Value When a Service is Created
State	Use this to enable or disable a service.	Enabled
Monitoring	Use this to enable service monitoring at a specific level or above. You can also use this to disable service monitoring for proxy services.	Disabled
Aggregation Interval	Use this to set the aggregation interval for the service.	10 minutes
SLA Alerts	Use this to enable SLA alerting for services at a specific level of severity or above. You can also use this to disable SLA alerting for a service.	Enabled
Pipeline Alerts	Use this to enable pipeline alerting for proxy services at a specific severity level or above. You can also use this to disable pipeline alerting for proxy services.	Enabled at Normal level or higher
Reports	Use this to enable or disable message reporting for proxy services.	Enabled at Normal level or higher
Logs	Use this to enable logging at a specific severity level or above. You can also use this to disable logging for proxy services. In proxy service log actions, the severity level of the log actions must match the Logging severity level on the proxy service operational settings. Note: To see log data in the log file or standard out (server console), Oracle WebLogic Server logging must be set to the following severity levels: Minimum severity to log: Info Log file: Info Standard out: Info For information on setting log severity levels, see "Using Log Severity Levels" in Oracle Fusion Middleware Configuring Log Files and Filtering Log Messages for Oracle WebLogic Server.	Enabled at Debug level or higher
Execution Tracing	Use this to enable or disable execution tracing for proxy services. Note: To see tracing in the log file or standard out (server console), Oracle WebLogic Server logging must be set to the following severity levels: Minimum severity to log: Info Log file: Info Standard out: Info For information on setting log severity levels, see "Using Log Severity Levels" in Oracle Fusion Middleware Configuring Log Files and Filtering Log Messages for Oracle WebLogic Server.	Disabled
Message Tracing	Use this to enable or disable message tracing for proxy services at a specific detail level or above. You can also set the payload limit (in kilobytes) and the default encoding. Note: See note for Execution Tracing.	Disabled
Offline Endpoint URIs	Use this to enable or disable non responsive endpoints for business services. You can also specify the interval of time to wait before retrying the offline endpoint URI. You can enable or disable offline URIs for business services only.	Disabled
Throttling State	Use this to enable or disable throttling for a business service.	Disabled
Maximum Concurrency	Use this to restrict the number of messages that can be concurrently processed by a business service.	0
Throttling Queue	Use this to restrict the maximum number of messages in the throttling queue.	0
Message Expiration	The maximum time interval (in milliseconds) for which a message can be placed in throttling queue.	0
Result Caching State	Use this to enable or disable result caching for a business service	Enabled Result caching is globally enabled by default

OSB VS BPEL, Which one to go for and Why...

OSB and oracle BPEL both are very popular tools for integrating the systems.

After going through some blogs, here is the quick info:

Basic Differences:

1. OSB is stateless and BPEL can be made stateless by setting the audit settings. However the audit settings turning off doesn't apply to mediator. 

2. OSB is far more superior in performance because it is stateless. In addition, it offers capabilities not supported in composite development such caching, throttling, and service virtualisation.

3. OSB has some severe limitations; the most serious being within transaction visibility, tracing, and metrics. It is either difficult, cumbersome, or impossible to find out if transactions have gone through the OSB engine and detailed, dynamic metrics are unavailable. 

4. OSB 11g cannot reference artifacts in the MDS (MetaData Store). Custom code is required for DVM support.

5. SOA Suite provides a better capability to handle transactions, compensation and asynchronous flows.

6. BPEL is better when the logic is complex as it has many inbuilt activities.

7. You dont get any monitoring and SLA management capability with Mediator and SOA. 

8. SOA Suite provides a better capability to handle transactions, compensation and Asynchronous.

9.As BPEL uses dehydration store to store the life cycle of a BPEL instance, it is very useful for long running processes.

Coming Back to OSB...

Few Features of OSB such as Service Result Caching, Message Throttling and service Virtualisation, makes it an ideal choice where Service Virtualization and 

 Message-Oriented Solutions are required.