Remove Duplicates From Fast

2 Useful links:
http://searchunleashed.wordpress.com/2011/12/08/how-remove-duplicate-results-works-in-fast-search-for-sharepoint/

http://msdn.microsoft.com/en-us/library/ff521593.aspx


Search a specific managed property:
and(string("hello world"), filter(property-spec:or(1, 20, 453, ... , 3473)))
and(string("hello world"), filter(property-spec:int("1 20 453 ... 3473", mode="or")))

Search all strings within specific managed property
and(string("."), filter(managedproperty-spec:or(variableofinput)))

 

Error on External List

Access Denied by Business Data Connectivity

After create Business Connectivity Service (BCS), I connect one of the web application to the service, then created an "External Content Type".  Finally created a list based on this external content type. The list, or the external list has an error when rendering on the web. The error message is "Access Denied by Business Data Connectivity".

To fix it, Go to Central Administration -> Application Management -> Manage Service Applications -> Business Data Connectivity Service* -> [Your Entity] -> Set Permissions, add the users who needs to access the newly created external type/list.

Can't Login

I was working on a server and we were trying to access the local SharePoint site http://127.0.0.1 or http://nameoflocalserver/pages/default.aspx and I was constantly prompted for the username and password. In SharePoint 2010, it can really annoy you by not accepting your username password credentials in the popup window.

There are two methods to do this:
1.  Specify the host names in the registry - BackConnectionHostNames (more secure and recommended for PRODUCTION servers). Refer http://support.microsoft.com/kb/896861
2. Disable the loopback check - DisableLoopbackCheck (less secure and recommended for DEVELOPMENT environments). Read on to use this method 2 and add via an easy powershell cmd.

use powershell:

New-ItemProperty HKLM:\System\CurrentControlSet\Control\Lsa -Name "DisableLoopbackCheck" -value "1" -PropertyType dword

Click Start, click Run, type regedit, and then click OK
In Registry Editor, locate the following registry key:HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
Right-click Lsa, point to New, and then click DWORD Value. (In Win 2008, its DWORD 32bit)
Type DisableLoopbackCheck, and then press ENTER.
Right-click DisableLoopbackCheck, and then click Modify.
In the Value data box, type 1 and then click OK.
Quit Registry Editor.
You may need to restart your server.

Invalid Property

Exception information:
Exception type: InvalidPropertyException
Exception message: Property doesn't exist or is used in a manner inconsistent with schema settings.

This maybe caused by the properties you add do not exist in your managed properties list.

Dynamic Rank

How Google Scores Fresh Content

Google Fellow Amit Singhal explains that “Different searches have different freshness needs.”

The implication is that Google measures all of your documents for freshness, then scores each page according to the type of search query. While some queries need fresh content, Google still uses old content for other queries (more on this later.)

Singhal describes the types of keyword searches most likely to require fresh content:

Recent events or hot topics: “occupy oakland protest” “nba lockout”
Regularly recurring events: “NFL scores” “dancing with the stars” “exxon earnings”
Frequent updates: “best slr cameras” “subaru impreza reviews”
Google’s patents offer incredible insight as to how web content can be evaluated using freshness signals, and rankings of that content adjusted accordingly.

Understand that these are not hard and fast rules, but rather theories consistent with patent filings, experiences of other SEOs, and experiments performed over the years. Nothing substitutes for direct experience, so use your best judgement and feel free to perform your own experiments based on the information below.

FAST Search for SharePoint 2010: Relevancy Tuning

Static rank points, also known as quality rank points, are added to search items before it is indexed. The number of points given is based on boost values that are stored in predefined static/quality rank components. You can decide which of these components to use at search time so that you only get the rank points from the components that you specify. You can control the effect of the static rank points by modifying the weight of the different components. Static rank points assigned to an indexed item are independent of the search words that are used, so static rank only indicates something about the general importance of an item. Adding static rank points is efficient from a search performance point of view because it does not add any complexity to the search evaluation.

Static rank points are derived from multiple managed properties. The following sets of managed properties are predefined for static ranking:

• Urldepthrank: Rank points given to boost shorter URLs.
• Docrank: Rank points given based on the number of and relative importance of links pointing to an item.
• Siterank: Rank points given based on the number of and relative importance of links pointing to the items on a site.
• Hwboost: FAST Search Server 2010 for SharePoint placeholder for generic usage of quality rank points.

In addition to these managed properties, you can add any custom integer managed property to the list of static/quality rank properties.

http://technet.microsoft.com/en-us/library/ff191247.aspx

Microsoft FAST Search Server 2010 offers a more powerful enterprise search for SharePoint 2010. Combining the power of FAST and simplicity of SharePoint, FAST Search Server 2010 offers an exceptional intranet and people search experience. It also serves as a platform for building search-driven applications.

The most powerful feature FAST offers for SharePoint is relevancy tuning. FAST Search Server 2010 comes with a predefined relevance out of the box and is tuned against a large set of various sample data. However, the data is not scoped for users, locations, organizations, keywords or content types. The search administrator has the option to create custom relevancy models tuned to differences in content sources, application needs and user contexts.

FAST Search assigns a relevancy score to each search index. Tuning relevancy manipulates the scoring system for the FAST Search system in the following ways.

• Static (quality) rank: Rank points added to a search item before it is indexed.
  The number of points given is based on boost values that are stored in four predefined static/quality rank components. In addition, any custom integer managed property can also be used to tune static rank. You can decide which of these components to use at search time so that you only get the rank points from the components that you specify. You can control the effect of the static rank points by modifying the weight of the different components. Static rank points assigned to an indexed item are independent of the search words used, so static rank only indicates something about the general importance of an item.
• Dynamic rank: Rank points added to search items at search time.
  The number of points given is based on a combination of the search words used and boost values that are stored in dynamic rank components. You can control the effect of the dynamic rank points by modifying the weight of the different components.
• Keyword rank: Rank points added to an item when a search word matches a specified keyword.
  Recall is improved by searching for the associated synonyms. Also, the search experience is improved by displaying additional relevant information with using Best Bets and Visual Best Bets. Also, incorporating document and site promotions to move relevant search results higher up in the search results list.
• Linguistic features: Like stemming, spell checking, stop words and tokenization effect both relevance and recall.
• Fast Query Language (FQL): A query language intended for creating queries programmatically.
  FQL can be used only for solutions developed for FAST Search Server 2010 for SharePoint.

The dynamic rank is more powerful than static rank because the ranking points are assigned at the search time based on search query. This makes it the most commonly used option for relevancy tuning. Dynamic rank option allows for boosting rank points in any of the following components of the indexed item.

• Freshness: Rank points are given based on age of item.
• Context: Rank points are given based on the search word hits in the item.
• Proximity: Rank points are given based on distance between the search words hit in the item.
• Managed Property Field Boost: Rank points are given based on value of managed property.
• Authority (anchor text): Rank points are given based on when a search word retrieves hits in the link text.
• Query Authority (click-through): Rank points are given based on when a search retrieves hits in items associated with previously clicked on search results.

As described herein, relevancy tuning allows an organization to implement a search solution that takes into account many variables that cause the most relevant results to rank highest in a search.

This tip contributed by Khanh Hoang, Abel Solutions’ Senior SharePoint Consultant.

Update Level

$mp = Get-FASTSearchMetadataManagedProperty -Name <ManagedProperty>
$mp.GetFullTextIndexMappings()
$fuu = Get-FASTSearchMetadataFullTextIndexMapping|Where-Object {$_.ManagedProperty.Name -eq “<ManagedProperty>”}
Remove-FASTSearchMetadataFullTextIndexMapping -Mapping $fuu
New-FASTSearchMetadataFullTextIndexMapping –ManagedProperty (Get-FASTSearchMetadataManagedProperty Body) –FullTextIndex (Get-FASTSearchMetadataFullTextIndex content) –Level <ImportanceLevel>
     

Update Context Weight, Remove Boost

$RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>
$content = $RankProfile.GetFullTextIndexRanks()|where-Object -filterscript {$_.FullTextIndexReference.Name -eq "content"}
$content.ContextWeight = <ContextWeight>
$content.Update()

$np = Get-FASTSearchMetadataRankProfile -Name $rankprofilename

#remove a ManagedPropertyBoosts
$np.GetManagedPropertyBoosts() | where-object -FilterScript {if($_.ManagedPropertyReference.Name -eq "productname") {$_.Delete()}}

#modify a ManagedPropertyBoosts value
$np.GetManagedPropertyBoosts() | where-object -FilterScript {if($_.ManagedPropertyReference.Name -eq "format") {$_.BoostValue="unknown format,-4000";$_.Update()}}
  
#add a boost
$RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>
$Property = Get-FASTSearchMetadataManagedProperty -Name <ManagedProperty>
$RankProfile.CreateManagedPropertyBoostComponent($Property, "<ManagedPropertyValue>,<ManagedPropertyBoostWeight>")
$RankProfile.Update()
#look up the value just added
$RankProfile.GetManagedPropertyBoosts()|where-object { $_.ManagedPropertyReference.Name -eq "<ManagedProperty>"}
 

Schema and Feature

The index schema and its features (FAST Search Server 2010 for SharePoint)
There are several reasons to for excluding the content of crawled properties to improve both relevance and recall. Some crawled properties might be searchable through a managed property mapping, and do not have to be searchable through the default full-text index also.
http://msdn.microsoft.com/en-us/subscriptions/gg982954.aspx


FAST Search Query Integration Overview
Using the Query Web Service
Using the Query Object Model
Restricting Search Results Using Search Scopes
http://msdn.microsoft.com/en-us/library/ff394628.aspx

FQL Language Syntax
http://msdn.microsoft.com/en-us/library/ff394462.aspx#fqloperator_filter

Tuning Dynamic Rank
http://technet.microsoft.com/en-us/library/ff453912.aspx

Managed Property and Rank

Managed Property
A managed property is content or metadata associated with an item that may be searched or used in other ways, such as being displayed in search results or used for query refinement. To make metadata searchable, first define the managed properties. Then, map the crawled properties that should be searchable to a managed property. At this point, the crawled metadata is searchable as a field. To search this metadata automatically as a general part of the item, map the content of a managed property into the default full-text index (known as content) for querying.
http://msdn.microsoft.com/en-us/subscriptions/gg982954.aspx

Ranking and Sorting (FAST Search Server 2010 for SharePoint)
Sort Search Results by Rank
If you want to use a rank profile that is different from the default rank profile, you can specify the name of the rank profile in the sort specification.
The RANK operator or XRANK operator in the FAST Query Language (FQL). These operators enable you to apply a conditional rank boosting if a specific query condition is met.

Sort Search Results by Managed Property Value
You can specify query result sorting based on the value of one or more managed properties in the query result. This means that FAST Search Server 2010 for SharePoint performs the sorting based on all results that match the query.
http://msdn.microsoft.com/en-us/library/ff394654.aspx#ranking_sorting_rank

Example:
criteria.OrderBy = new List<OrderData>()
{
new OrderData(new RankPropertyExpression("departments"),
OrderDirection.Ascending)
};
http://documentation.ektron.com/cms400/v8.60/Reference/Web/Navigating/Search/Setting_up_FAST.htm

Change Importance Level


$RankProfile = Get-FASTSearchMetadataRankProfile -Name default1
$content = $RankProfile.GetFullTextIndexRanks()|where-Object -filterscript {$_.FullTextIndexReference.Name -eq "content"}
$content.SetImportanceLevelWeight(1, 5)
$content.Update()
$content.GetImportanceLevelWeight(1)

http://fs4sp.blogspot.com/2012_01_01_archive.html

Fast Search Refinement and Scope

Manage refiners and refiner settings (FAST Search Server 2010 for SharePoint)
http://technet.microsoft.com/en-us/library/gg193929.aspx
Set-FASTSearchMetadataManagedProperty -Name <Name> -RefinementEnabled <Flag>
To enable a managed property as shallow refiner by using Windows PowerShell
$mp = Get-FASTSearchMetadataManagedProperty –Name <ManagedProperty>
$refiner = $mp.GetRefinerConfiguration()
$refiner.RefinementType = “DeepRefinementDisabled”
$mp.SetRefinerConfiguration($refiner)
$refiner
Use Powershell to create site scope
Search scopes in FAST Search for SharePoint (Part 1)
http://blogs.msdn.com/b/jorgeni/archive/2010/02/26/search-scopes-in-fast-search-for-sharepoint-part-1.aspx
New-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Name MySimpleScope -Description "My Simple Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter ‘title:vista’
Specifying which fulltext index to search
FAST Search scope filters have one additional capability; you can in the filter specify which non-default fulltext index (=composite field if you’re familiar with FAST ESP terminology) you want to query. To specify the fulltext index as part of, or the whole scope filter use “FullTextIndex=” as part of the filter. Note that the you separate the FullTextIndex parameter and the value with equals (=).
New-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Name MySimpleScope -Description "My Simple Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter ‘title:vista,FullTextIndex= mycustomfulltextindex’

Fast Search Query Tool
http://fastforsharepoint.codeplex.com/

Fast Search

Using FQL to query FAST Search Index
http://blogs.perficient.com/microsoft/2011/06/using-fql-to-query-fast-search-index/
Write a sample project, use FQL syntax to query FAST Search server through search services.
Can also use FQL on KeywordQuery Class build a console application to test FQL syntax.

FastSearch and PowerShell, use powershell to create content source, start crawl, execute search.
https://www.nothingbutsharepoint.com/sites/itpro/Pages/Fast-Search-Server-2010-for-SharePoint-Proof-of-Concept-Part-2b-PowerShell-Configuring-the-connector-Indexing-Content.aspx

Change Managed Property boost using PowerShell
$RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>
$Property = Get-FASTSearchMetadataManagedProperty -Name <ManagedProperty>
$RankProfile.CreateManagedPropertyBoostComponent($Property, "<ManagedPropertyValue>,<ManagedPropertyBoostWeight>")
$RankProfile.Update()
http://technet.microsoft.com/en-us/library/ff191224.aspx


Change the overall- and managed property context weight by using Windows PowerShell (FAST Search Server 2010 for SharePoint)
http://technet.microsoft.com/en-us/library/ff191252.aspx

Fast Search Articles
http://gallery.technet.microsoft.com/office/site/search?f%5B0%5D.Type=User&f%5B0%5D.Value=Jan%20Inge%20Bergseth
Create a custom full text index using powershell
http://gallery.technet.microsoft.com/office/Create-a-custom-full-text-14811b11
Create and tune a custom rank profile for fastsearch
http://gallery.technet.microsoft.com/office/Create-and-tune-a-custom-c2f79c49
Manage Crawled Properties
http://technet.microsoft.com/en-us/library/ff191246.aspx
exclude a crawled property from indexing
$CrawledProperty = Get-FASTSearchMetadataCrawledProperty -Name "<CrawledPropertyName>"
Set-FASTSearchMetadataCrawledProperty -CrawledProperty $CrawledProperty -IsMappedToContents $false
indentify unmapped crawled properites
$category = Get-FASTSearchMetadataCategory -Name "<CategoryName>"
$unmappedCPs = $category.GetUnmappedCrawledProperties()

Good aritcle on index schema and its features
The index schema and its features (FAST Search Server 2010 for SharePoint)
http://technet.microsoft.com/en-us/library/gg982954.aspx