Inside the Force.com Query Optimizer Webinar

Inside the Force.com Query Optimizer
From salesforce.com’s Customer Centric Engineering – Technical Enablement team

Join the conversation: #forcewebinar
Safe harbor
Safe harbor statement under the Private Securities Litigation Reform Act of 1995:
This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties
materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results
expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be
deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other
financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any
statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services.
The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new
functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our
operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of
intellectual property and other litigation, risks associated with possible mergers and acquisitions, the immature market in which we
operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new
releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization
and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com,
inc. is included in our annual report on Form 10-Q for the most recent fiscal quarter ended July 31, 2012. This documents and others
containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site.
Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently
available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based
upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-
looking statements.

Join the conversation: #forcewebinarJoin the conversation: #forcewebinar
John Tan
Architect Evangelist
@johntansfdc
Jaikumar Bathija
Architect – DB Performance
@
Speakers

Follow Developer Force for the latest news
@forcedotcom / #forcewebinar
Developer Force group
Developer Force – Force.com Community
+Developer Force – Force.com Community
Developer Force

Architect Core Resource page
•  Featured content for architects
•  Articles, papers, blog posts, events
•  Follow us on Twitter
Updated weekly!
http://developer.force.com/architect

Have questions?
§  We have an expert support team at the ready to answer your questions
during the webinar.
§  Ask your questions via the GoToWebinar Questions Pane.
§  The speaker(s) will choose top questions to answer live at the end of the
webinar.
§  Please post your questions as we go along!
§  Only post your question once; we’ll get to it as we go down the list.

Today s Learning Goal
AWARENESS

Why are we here?
SELECT Id
FROM Account
WHERE Status__c != ‘Closed’ AND
Rating = Null AND
CreatedDate > 2013-04-01
Empower developers to write selective queries. Don’t worry we have lots of examples.

Selective Filters
Reduces the
number of
records in your
result set.
Leverages
indexes.
Avoids full table
scans.

Query Performance Impact
User Experience
(Visualforce
pages, API,
Reports,
Listviews, etc).
Governor Limits
(Timeouts,
Concurrent
Request Limit,
Concurrent API
limit, etc).
Large Data
Volumes (LDV).

Agenda
•  Design - http://developer.force.com/architect
•  Query Optimizer
•  SOQL Examples
•  Skinny Tables
•  Other Performance Factors

Query Optimizer

Query execution

Multitenancy

Basic Algorithm
•  Pre-Query engine.
•  Chooses the most selective filter from the WHERE clause.
•  Determine the best leading table/index to drive the query.

Indexing
•  Standard index - is available out of the box and we have a whole bunch of
fields that are indexed on Standard and custom entities.
•  Custom index – is created on-demand, based on performance analysis done
pro-actively by salesforce team.
•  What other fields are indexed – External Id fields, fields marked unique,
foreign keys by way of lookup or master detail relationship.

Statistics
•  Pre-computed Statistics
§  Row count
§  User visibility
§  Custom index
§  Owner row count

Options considered by the Optimizer

The numbers game
•  Standard index will be considered only if the filter fetches < 30% of the
records for the first million records and less than 15% of the records after the
first million records, up to 1M records. * The selectivity threshold is subject
to change.
•  Custom index will be considered only if the filter fetches < 10% of the records
for the first million records and less than 5% of the records after the first
million records, up to 333,333 records. * The selectivity threshold is
subject to change.

The numbers game – Standard Index
# of records First Threshold Second Threshold Final Threshold
Up to 1 million 30% of total N/A 30% of total
Up to 2 million 300,000 150,000 450,000
Up to 3 million 300,000 300,000 600,000
Up to 4 million 300,000 450,000 750,000
Up to 5 million 300,000 600,000 900,000
Above 5.6 million 300,000 700,000 1,000,000

The numbers game – Custom Index
# of records First Threshold Second Threshold Final Threshold
Up to 1 million 10% of total N/A 10% of total
Up to 2 million 100,000 50,000 150,000
Up to 3 million 100,000 100,000 200,000
Up to 4 million 100,000 150,000 250,000
Up to 5 million 100,000 200,000 300,000
Above 5.6 million 100,000 233,333 333,333

Other Optimizations
AND optimizations
§  Composite Index Join - INTERSECTION of indexes should still meet
selectivity threshold.
OR optimizations
§  Union - SUM of the filters should still meet selectivity threshold.
sort optimizations
§  an index aligns with our order by clause and the query has a row limit,
we can use the index to find the first rows quickly and exit.

Examples

Schema
MyCase – 100,000 Records
MyUser – 100,000 Records

Reminder: Goal of Optimizer
•  Generate efficient SQL
•  Leverage an index to drive query
•  Avoid full table scans
Query Optimizer cannot make up for non-selective filters. It will
make the best choice from the filters in your query.

Selectivity
SELECT Id FROM MyCase__c
WHERE Status__c = ‘Closed’ will be do a full table scan
WHERE Status__c = ‘New’ will use the index
Indexed Field Value # of Records Selective?
Status Closed 96,500 No
Status New 3,500 Yes
Total # Records = 100,000
Selectivity Threshold = 10,000

Not Equals / Not In
Can’t use index because of not equals
WHERE Priority__c != 3 will do a full table scan
WHERE Priority__c IN (1,2) will use the index
Indexed Value # of Records Selective?
Priority 1 6,000 Yes
Priority 2 3,500 Yes
Priority 3 90,500 No

Formula Fields
Field Type Formula
CaseType__c Formula
(Text)
CASE(MyUser__r.UserType__c,1,”Gold”,”Silver”)
Can’t create an index on CaseType__c since this formula spans objects
IF MyUser__r.UserType__c has an index
•  SELECT Id FROM MyCase__c WHERE MyUser__r.UserType__c = 1

Formula Fields
Field Type
CaseTypeClone__c Text(255)
Or avoid a join and create CaseTypeClone__c field and index it
•  SELECT Id FROM MyCase__c WHERE CaseTypeClone__c = ‘Gold’

Indexed Field Value # of Records Selective?
ClosedDate Non-Null 96,500 Yes for specific dates
ClosedDate Null 3,500 Yes
Nulls
Customer Support will need to create a custom index that includes null records. Standard indexes by default
include nulls.
SELECT Id FROM MyCase__c WHERE ClosedDate__c = null will use the index
http://blogs.developerforce.com/engineering/2013/02/force-com-soql-best-practices-nulls-and-formula-fields.html

Date & Number Range
SELECT Id
FROM MyCase
WHERE ClosedDate__c > 2013-01-01 AND ClosedDate__c < 2013-02-01
Query Optimizer can detect only date and number ranges.

AND conditions
Composite Index Join
SELECT Id FROM MyUser
WHERE FirstName__c = ‘Jane’ AND LastName__c = ‘Doe’ AND City__c = ‘San Francisco’
Step 1 – Allow each index to still be considered if they return < 2X selectivity threshold
Step 2 – INTERSECTION of all indexes must meet *selectivity threshold
Step 3 – Use composite index join to drive query
*If all indexes are standard indexes, use standard index selectivity threshold. Otherwise, use the custom index standard selectivity
threshold

AND conditions
Composite Index Join – MyUser object 100,000 records

2-column Index
For this simple example, it makes more sense to have Customer Support create a 2-column
index.

OR conditions
Union
SELECT Id FROM MyUser
WHERE FirstName__c = ‘Jane’ OR LastName__c = ‘Doe’ OR City__c = ‘San Francisco’
Step 1 – Each field must be indexed and meet selectivity threshold
Step 2 – ADDITION of all the indexes must meet *selectivity threshold
Step 3 – Use union to drive query
*If all indexes are standard indexes, use standard index selectivity threshold. Otherwise, use the custom index standard selectivity
threshold

OR conditions
Union – MyUser object 100,000 records

OR conditions
Using SOSL may be a better option
•  SELECT Id FROM Account WHERE PersonMobilePhone LIKE ‘%123’ – leading %
wildcard as bad as full scan
•  SELECT Id FROM Account WHERE PersonMobilePhone = ‘1234567890’ OR
PersonHomePhone = ‘1234567890’ OR Phone = ‘1234567890’

Relationship
Relationship
WHERE MyUser__r.JobType = 1 AND Priority__c = ‘Priority 1’
Each index’s selectivity threshold is analyzed separately and the index with the lower
threshold % is chosen.

Soft Deletes
•  Records in the Recycle Bin with isDeleted = true
•  DO NOT USE isDeleted = false as a filter
•  Counted in pre-computed statistics
•  Use hard delete option in Bulk API or Contact Customer Support

Sort Optimization
•  Number and Date fields only
•  Limit Clause required
•  Can make up for a non-selective filter
ORDER BY CreatedDate LIMIT 10
WHERE CreatedDate > 2001-01-01
ORDER BY CreatedDate LIMIT 10

Review
ü  Selectivity thresholds determine if an index is considered
ü  Not Equals filters will not leverage indexes
ü  Be careful filtering on Null
ü  And conditions involve an INTERSECTION of indexes
ü  OR conditions involve an ADDITION of indexes
ü  ORDER BY with a LIMIT on an index can make up for non-selective filters

Sharing

Record Visibility
•  Applies only to non-Admin users.
•  Depending on your user profile, you may have visibility to few or large
number of records.
•  Sharing tables may drive query instead of index

Skinny Table

Skinny Tables
•  Single Object
•  Maximum of 100 fields
•  Not Aggregate/Summary. 1:1 record count between source object and skinny
•  It is not a cross-object join
•  Updates to source object automatically reflected in skinny
•  Improved performance – minimal joins since fields are in one table

When are Skinny Tables used?
ü  After attempting to tune with custom indexes
ü  All fields selected and filtered must be in skinny
ü  Salesforce.com will analyze and create

Other Performance Factors

Performance Factors
Sharing
§  Test as a non-System Admin User
Data Skews
§  Avoid parent-child and ownership data skews
Archiving
Database Caching
§  Avoid relying on cache performance or attempting to warm the cache

Key Takeaways
ü  Query Performance improves with indexes
ü  Use selective filters to reduce result set
ü  Query Optimizer chooses the best table/index to drive a query
ü  Skinny Tables may help when indexing is exhausted

Cheat Sheet:
Indexed Fields
http://developer.force.com/architect
Query & Search Optimization
Cheat Sheet
Database
http://developer.force.com
Query Optimization Overview
When building queries, list views, and reports, it's best to create filter conditions
that are selective so that Force.com scans the most appropriate rows in the
objects that your queries target. This best practice is especially important when
your queries target "large objects," objects containing more than one million
records.
When writing SOQL, consider using the following fields, which can make your
query filter conditions more selective, and improve your query response times
and your database's overall performance.
Selectivity Overview
Several things can affect the selectivity of a query filter's conditions.
Whether the field in the condition has an index
Whether the value in the condition is selective relative to the total number of records in the
object. These numbers determine the selectivity threshold, which the Force.com query optimizer
uses to ensure that the most appropriate index, if any, drives each of your queries.
Whether the operator in the condition permits the use of available indexes
When writing your queries, remember the following selectivity conditions and tips.
SOQL
Fields with Database Indexes
Primary
Keys
Foreign
Keys
Audit
Dates
Custom
Fields
Id
Name
OwnerId
CreatedById
LastModifiedById
Lookup fields
Master-detail
relationship fields
CreatedDate
LastActivityDate
SystemModstamp
Unique fields
External ID
fields
Index Selectivity Conditions and Thresholds
Unary Condition:
Standard Index
Unary Condition:
Custom Index
AND
Condition
OR
Condition
LIKE
Condition
Force.com uses a
standard index if
the filter targets less
than:
30% of the first
million records
15% of all records
after the first million
records
1 million total
records
Force.com uses a
custom index if the
filter targets less
than:
10% of the first
million records
5% of all records
after the first
million records
333,333 total
records
Force.com uses
a composite
index join if the
filter targets less
than:
Twice the index
selectivity
thresholds for
each field
The index
selectivity
thresholds for
the intersection
of those fields
Force.com
uses a union
if the filter
targets less
than:
The index
selectivity
thresholds for
each field
The index
selectivity
thresholds for
the sum of
those fields
For
conditions
that don't
start with
a leading
wildcard,
Force.com
tests the
first 100,000
rows for
selectivity.
Query Optimization Resources
In addition to this cheat sheet's previous sections, we recommend reading the
following related resources, which can help you retrieve the records you want
from a large volume of data—and do so quickly and efficiently.
Best Practices for Deployments with Large Data Volumes (white paper)
Force.com Apex Code Developer's Guide (guide)
Force.com Blogs: Engineering (blog posts)
How to Improve Listview Performance (Salesforce Knowledge article)
In the online help:
» "Build Effective Filters"
» "Getting the Most Out of Filter Logic"
» "Improve Report Performance"
Index Selectivity Exceptions
When you build a filter condition with the following operators, Force.com doesn't use an available
index. Instead, it scans all records in the object to find the records that satisfy the condition. Feel
free to use these operators, but be sure to add selective filter conditions.
The following filter operators
» not equal to
» contains
» does not contain
When used with text and text fields, the following comparison operators
» (<)
» (>)
» (<=)
» (>=)
Additionally, Force.com doesn't use available indexes when you use:
Leading wildcards
Non-deterministic or cross-object formula fields
SOSL
Fields with Search Indexes Search Selectivity Tips
General Sidebar Search and Advanced Search
Be as selective as possible. For
example, use Michael*, not Mich*.
Remember that Chatter feed
searches aren't affected by the scope
of your search; Chatter feed search
results include matches across all
objects.
Search for the exact phrase with an advanced search.
Limit scope by targeting:
» Specific objects
» Rows owned by the searcher
» Rows within a division, when applicable
See "Search Overview" in the online help.
General
Name fields
Phone fields
Text fields
Picklist fields
These fields vary by object. See "Search Fields" in the online help.

Upcoming Events
April 21-27, 2013
Salesforce Mobile Developer Week
May 8, 2013
Summer ‘13 Release Developer Preview Webinar
May 9, 2013
SOQL Best Practices CodeTalk

Survey
Your feedback is crucial to the success of our webinar programs.
Thank you!
http://bit.ly/querysurvey
*Look in the GoToWebinar chat
window now for a hyperlink.

John Tan
Architect Evangelist
@johntansfdc
Jaikumar Bathija
Architect – DB Perfomance
Q&A

Inside the Force.com Query Optimizer Webinar

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Inside the Force.com Query Optimizer Webinar

Similar to Inside the Force.com Query Optimizer Webinar (20)

More from Salesforce Developers

More from Salesforce Developers (20)

Recently uploaded

Recently uploaded (20)

Inside the Force.com Query Optimizer Webinar