OnBase Troubleshooting Greg Wheeler, Customer Care Team Manager What went wrong? Something is misconfigured Bad data causes recurring errors Your specific environment causes a strange problem with the software A performance bottleneck causes errors Custom Code is failing You may have discovered a bug in the software Two sample scenarios Documents stopped appearing in a Workflow queue “Problem with the environment” Unity Client error/ performance troubleshooting “Problem with the software’s behavior” (based on actual Customer Care Team solved cases) Scenario #1: Where are my files? Users report that a Workflow queue that normally contains documents is empty These Files are images that are released from another system Just started this morning! Uh, where are they? ? Where do we start? Most problems are way too big—we need to make them smaller and more manageable/testable Data Collection We need to know what the user, application, script or service was doing We need to try to differentiate between a coding, data or environmental cause Can it be reproduced at will? What about in another environment? What do the logs say? Narrowing an issue down Is it a problem with thick client, Core Services, or both? What servers can we take out of the equation? What modules can we eliminate? What functionality can we exclude? Diskgroup Storage (SAN) OnBase Thick Client (Workflow TImers) Unity Client Application Server OnBase Thick Client (DIP) Database Server SMB/CIFS Diskgroup Storage (SAN) OnBase Thick Client (Workflow TImers) HTTP Unity Client ODBC Application Server OnBase Thick Client (DIP) Database Server First Step: Is Workflow…working? Is it limited to this queue/lifecycle in Workflow, or are there others affected? ? Diskgroup Storage OnBase Thick Client (Workflow TImers) Application Server OnBase Thick Client (DIP) Database Server Thoughts We see timers are working – We see other documents moving around We can look at the timer machine to make sure THIS timer is still running Let’s trace further back Is DIP…you know, DIPping? Diskgroup Storage OnBase Thick Client (Workflow TImers) ? Application Server OnBase Thick Client (DIP) Database Server What to check? DIP is importing documents as a service on the DIP machine To troubleshoot DIP, we would stop the service, then fire up a client with the SCHED switch If you see this…it’s not OnBase If you see this…there’s a problem Where did it fail? Sys – Verification Reports Let’s look at the directory ProcMon Demo Process Monitor ProcMon can be used to see behind-the-scenes Windows calls and access requests http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx Solution Issue #2: Unity Performance Users are experiencing slow client performance first thing every morning Workflow queues taking a long time to navigate between documents Sometimes happens in custom queries, document retrieval All users affected SMB/CIFS Diskgroup Storage OnBase Thick Client (Workflow TImers) HTTP Unity Client ODBC Application Server OnBase Thick Client (DIP) Database Server SMB/CIFS Diskgroup Storage HTTP Unity Client ODBC Application Server Database Server Options Observe Unity directly on Application Server (still slow?) What about testing the thick client? (Verbose) Look at Application Server performance (perfmon…or resmon) Look at connectivity to DB/Diskgroups (ping/WireShark) Look at Diagnostics Console Let’s run Unity from the AppServer SMB/CIFS Diskgroup Storage ODBC Application Server Database Server Sometimes Thick Client is valid… Test the Database Choose a SELECT query and run it directly on the database server. Compare times to complete Is the query coming back quickly enough? Perfmon Consider ResMon Diagnostics Console - Service Tab Change to “True” What can ResMon tell you? Test Connectivity Application Server > Database Application Server > Diskgroup Diagnostics Console “What went wrong?” “What is OnBase doing right now?” “What queries are running, and how fast?” Advice Beware the Database tab in Production Consider logging to a file to review on another machine Log the least amount of information needed to troubleshoot The rest of the story Grabbed some queries from the Database tab from around the same time Ran them in SQL Server Management Studio Slow there too! Unoptimized indexes The Smoking Gun Your Issues!